使用 GPT 文本嵌入和 CLIP 图像嵌入对维基百科文章进行多模态检索¶

在本 notebook 中，我们将演示如何使用 LlamaIndex 构建一个多模态检索系统。

维基百科文本嵌入索引：使用 OpenAI 生成文本的 GPT 文本嵌入

维基百科图像嵌入索引：使用 OpenAI 生成图像的 CLIP 嵌入

查询编码器

• 使用 GPT 嵌入为文本索引编码查询文本
• 使用 CLIP 嵌入为图像索引编码查询文本

框架：LlamaIndex

步骤

1. 下载维基百科文章的文本和图像原始文件
2. 使用 GPT 嵌入为向量存储构建文本索引
3. 使用 CLIP 嵌入为向量存储构建图像索引
4. 使用不同的查询编码嵌入和向量存储同时检索相关的文本和图像

In [ ]

%pip install llama-index-vector-stores-qdrant

%pip install llama-index-vector-stores-qdrant

In [ ]

%pip install llama_index ftfy regex tqdm
%pip install git+https://github.com/openai/CLIP.git
%pip install torch torchvision
%pip install matplotlib scikit-image
%pip install -U qdrant_client

%pip install llama_index ftfy regex tqdm %pip install git+https://github.com/openai/CLIP.git %pip install torch torchvision %pip install matplotlib scikit-image %pip install -U qdrant_client

加载和下载包含维基百科文本和图像的多模态数据集¶

解析维基百科文章并保存到本地文件夹

In [ ]

from pathlib import Path
import requests

wiki_titles = [
    "batman",
    "Vincent van Gogh",
    "San Francisco",
    "iPhone",
    "Tesla Model S",
    "BTS",
]


data_path = Path("data_wiki")

for title in wiki_titles:
    response = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query",
            "format": "json",
            "titles": title,
            "prop": "extracts",
            "explaintext": True,
        },
    ).json()
    page = next(iter(response["query"]["pages"].values()))
    wiki_text = page["extract"]

    if not data_path.exists():
        Path.mkdir(data_path)

    with open(data_path / f"{title}.txt", "w") as fp:
        fp.write(wiki_text)

from pathlib import Path import requests wiki_titles = [ "batman", "Vincent van Gogh", "San Francisco", "iPhone", "Tesla Model S", "BTS", ] data_path = Path("data_wiki") for title in wiki_titles: response = requests.get( "https://en.wikipedia.org/w/api.php", params={ "action": "query", "format": "json", "titles": title, "prop": "extracts", "explaintext": True, }, ).json() page = next(iter(response["query"]["pages"].values())) wiki_text = page["extract"] if not data_path.exists(): Path.mkdir(data_path) with open(data_path / f"{title}.txt", "w") as fp: fp.write(wiki_text)

解析维基百科图像和文本。加载到本地文件夹¶

In [ ]

import wikipedia
import urllib.request

image_path = Path("data_wiki")
image_uuid = 0
# image_metadata_dict stores images metadata including image uuid, filename and path
image_metadata_dict = {}
MAX_IMAGES_PER_WIKI = 30

wiki_titles = [
    "San Francisco",
    "Batman",
    "Vincent van Gogh",
    "iPhone",
    "Tesla Model S",
    "BTS band",
]

# create folder for images only
if not image_path.exists():
    Path.mkdir(image_path)


# Download images for wiki pages
# Assing UUID for each image
for title in wiki_titles:
    images_per_wiki = 0
    print(title)
    try:
        page_py = wikipedia.page(title)
        list_img_urls = page_py.images
        for url in list_img_urls:
            if url.endswith(".jpg") or url.endswith(".png"):
                image_uuid += 1
                image_file_name = title + "_" + url.split("/")[-1]

                # img_path could be s3 path pointing to the raw image file in the future
                image_metadata_dict[image_uuid] = {
                    "filename": image_file_name,
                    "img_path": "./" + str(image_path / f"{image_uuid}.jpg"),
                }
                urllib.request.urlretrieve(
                    url, image_path / f"{image_uuid}.jpg"
                )
                images_per_wiki += 1
                # Limit the number of images downloaded per wiki page to 15
                if images_per_wiki > MAX_IMAGES_PER_WIKI:
                    break
    except:
        print(str(Exception("No images found for Wikipedia page: ")) + title)
        continue

import wikipedia import urllib.request image_path = Path("data_wiki") image_uuid = 0 # image_metadata_dict stores images metadata including image uuid, filename and path image_metadata_dict = {} MAX_IMAGES_PER_WIKI = 30 wiki_titles = [ "San Francisco", "Batman", "Vincent van Gogh", "iPhone", "Tesla Model S", "BTS band", ] # create folder for images only if not image_path.exists(): Path.mkdir(image_path) # Download images for wiki pages # Assing UUID for each image for title in wiki_titles: images_per_wiki = 0 print(title) try: page_py = wikipedia.page(title) list_img_urls = page_py.images for url in list_img_urls: if url.endswith(".jpg") or url.endswith(".png"): image_uuid += 1 image_file_name = title + "_" + url.split("/")[-1] # img_path could be s3 path pointing to the raw image file in the future image_metadata_dict[image_uuid] = { "filename": image_file_name, "img_path": "./" + str(image_path / f"{image_uuid}.jpg"), } urllib.request.urlretrieve( url, image_path / f"{image_uuid}.jpg" ) images_per_wiki += 1 # Limit the number of images downloaded per wiki page to 15 if images_per_wiki > MAX_IMAGES_PER_WIKI: break except: print(str(Exception("找不到维基百科页面的图片：")) + title) continue

In [ ]

import os

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

import os os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

在不同集合下使用文本和图像嵌入构建多模态向量存储¶

In [ ]

import qdrant_client
from llama_index.core import SimpleDirectoryReader
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.indices import MultiModalVectorStoreIndex

# Create a local Qdrant vector store
client = qdrant_client.QdrantClient(path="qdrant_db")

text_store = QdrantVectorStore(
    client=client, collection_name="text_collection"
)
image_store = QdrantVectorStore(
    client=client, collection_name="image_collection"
)
storage_context = StorageContext.from_defaults(
    vector_store=text_store, image_store=image_store
)

# Create the MultiModal index
documents = SimpleDirectoryReader("./data_wiki/").load_data()
index = MultiModalVectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
)

import qdrant_client from llama_index.core import SimpleDirectoryReader from llama_index.vector_stores.qdrant import QdrantVectorStore from llama_index.core import VectorStoreIndex, StorageContext from llama_index.core.indices import MultiModalVectorStoreIndex # Create a local Qdrant vector store client = qdrant_client.QdrantClient(path="qdrant_db") text_store = QdrantVectorStore( client=client, collection_name="text_collection" ) image_store = QdrantVectorStore( client=client, collection_name="image_collection" ) storage_context = StorageContext.from_defaults( vector_store=text_store, image_store=image_store ) # Create the MultiModal index documents = SimpleDirectoryReader("./data_wiki/").load_data() index = MultiModalVectorStoreIndex.from_documents( documents, storage_context=storage_context, )

/Users/haotianzhang/llama_index/venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

绘制从维基百科下载的图像¶

In [ ]

from PIL import Image
import matplotlib.pyplot as plt
import os


def plot_images(image_metadata_dict):
    original_images_urls = []
    images_shown = 0
    for image_id in image_metadata_dict:
        img_path = image_metadata_dict[image_id]["img_path"]
        if os.path.isfile(img_path):
            filename = image_metadata_dict[image_id]["filename"]
            image = Image.open(img_path).convert("RGB")

            plt.subplot(8, 8, len(original_images_urls) + 1)
            plt.imshow(image)
            plt.xticks([])
            plt.yticks([])

            original_images_urls.append(filename)
            images_shown += 1
            if images_shown >= 64:
                break

    plt.tight_layout()


plot_images(image_metadata_dict)

from PIL import Image import matplotlib.pyplot as plt import os def plot_images(image_metadata_dict): original_images_urls = [] images_shown = 0 for image_id in image_metadata_dict: img_path = image_metadata_dict[image_id]["img_path"] if os.path.isfile(img_path): filename = image_metadata_dict[image_id]["filename"] image = Image.open(img_path).convert("RGB") plt.subplot(8, 8, len(original_images_urls) + 1) plt.imshow(image) plt.xticks([]) plt.yticks([]) original_images_urls.append(filename) images_shown += 1 if images_shown >= 64: break plt.tight_layout() plot_images(image_metadata_dict)

在不同的集合 wikipedia_img 下构建一个单独的 CLIP 图像嵌入索引¶

In [ ]

def plot_images(image_paths):
    images_shown = 0
    plt.figure(figsize=(16, 9))
    for img_path in image_paths:
        if os.path.isfile(img_path):
            image = Image.open(img_path)

            plt.subplot(2, 3, images_shown + 1)
            plt.imshow(image)
            plt.xticks([])
            plt.yticks([])

            images_shown += 1
            if images_shown >= 9:
                break

def plot_images(image_paths): images_shown = 0 plt.figure(figsize=(16, 9)) for img_path in image_paths: if os.path.isfile(img_path): image = Image.open(img_path) plt.subplot(2, 3, images_shown + 1) plt.imshow(image) plt.xticks([]) plt.yticks([]) images_shown += 1 if images_shown >= 9: break

获取一些示例查询的多模态检索结果¶

In [ ]

test_query = "who are BTS team members"
# generate  retrieval results
retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5)
retrieval_results = retriever.retrieve(test_query)

test_query = "who are BTS team members" # generate retrieval results retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5) retrieval_results = retriever.retrieve(test_query)

In [ ]

from llama_index.core.response.notebook_utils import display_source_node
from llama_index.core.schema import ImageNode

retrieved_image = []
for res_node in retrieval_results:
    if isinstance(res_node.node, ImageNode):
        retrieved_image.append(res_node.node.metadata["file_path"])
    else:
        display_source_node(res_node, source_length=200)

plot_images(retrieved_image)

from llama_index.core.response.notebook_utils import display_source_node from llama_index.core.schema import ImageNode retrieved_image = [] for res_node in retrieval_results: if isinstance(res_node.node, ImageNode): retrieved_image.append(res_node.node.metadata["file_path"]) else: display_source_node(res_node, source_length=200) plot_images(retrieved_image)

节点 ID: e30e1817-4e31-4047-be5d-37502560920c
相似性 0.808149809808292
文本: BTS (Korean: 방탄소년단; RR: Bangtan Sonyeondan; lit. Bulletproof Boy Scouts)，也称为 Bangtan Boys，是一个于 2010 年成立的韩国男子乐队。乐队由 Jin, Suga, J-Hope, RM, Jimi...

节点 ID: 024f3296-37c8-46d5-a184-2f78c621a99f
相似性 0.7987048642063129
文本: === 粉丝群 === 根据 Kyung Hyun Kim 的说法，BTS 的崛起得益于 YouTube 上音乐视频节目制作和消费的大幅增长以及偶像帝国的到来，包括周边...

节点 ID: c564ccf4-a94f-408f-8b21-224538dc2e94
相似性 0.7838098925118134
文本: == 历史 ==

文本: === 2010–2014: 成立和早期 === BTS 于 2010 年成立，此前 Big Hit Entertainment 首席执行官 Bang Si-hyuk 希望围绕地下说唱歌手 RM（金南俊）组建一个嘻哈组合...

In [ ]

test_query = "what are Vincent van Gogh's famous paintings"
# generate  retrieval results
retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5)
retrieval_results = retriever.retrieve(test_query)

retrieved_image = []
for res_node in retrieval_results:
    if isinstance(res_node.node, ImageNode):
        retrieved_image.append(res_node.node.metadata["file_path"])
    else:
        display_source_node(res_node, source_length=200)

plot_images(retrieved_image)

test_query = "what are Vincent van Gogh's famous paintings" # generate retrieval results retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5) retrieval_results = retriever.retrieve(test_query) retrieved_image = [] for res_node in retrieval_results: if isinstance(res_node.node, ImageNode): retrieved_image.append(res_node.node.metadata["file_path"]) else: display_source_node(res_node, source_length=200) plot_images(retrieved_image)

节点 ID: e002927c-0bf5-482b-a0a1-0ee2f3cd48f9
相似性 0.8675476190545354
文本: 文森特·威廉·梵高（荷兰语：[ˈvɪnsɛnt ˈʋɪləɱ vɑŋ ˈɣɔx]；1853 年 3 月 30 日 – 1890 年 7 月 29 日）是一位荷兰后印象派画家，是历史上最著名和最有影响力的人物之一...

节点 ID: 69ef1c64-a5b4-468c-a58c-7d36151961a7
相似性 0.8661792475490765
文本: ==== 花卉 ====

文本: 梵高创作了多幅花卉风景画，包括玫瑰、丁香、鸢尾花和向日葵。其中一些反映了他对色彩语言的兴趣，以及对日本浮世绘...

节点 ID: f971a611-a8b9-48b4-a81b-d3856438aab8
相似性 0.8616832203971132
文本: ==== 肖像画 ====

文本: 梵高说肖像画是他最大的兴趣。他写道：“我最热情的东西，远远超过我职业中的其他一切，”他在 1890 年写道，“是肖像画，它...

In [ ]

test_query = "what is the popular tourist attraction in San Francisco"
# generate  retrieval results
retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5)
retrieval_results = retriever.retrieve(test_query)

retrieved_image = []
for res_node in retrieval_results:
    if isinstance(res_node.node, ImageNode):
        retrieved_image.append(res_node.node.metadata["file_path"])
    else:
        display_source_node(res_node, source_length=200)

plot_images(retrieved_image)

test_query = "what is the popular tourist attraction in San Francisco" # generate retrieval results retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5) retrieval_results = retriever.retrieve(test_query) retrieved_image = [] for res_node in retrieval_results: if isinstance(res_node.node, ImageNode): retrieved_image.append(res_node.node.metadata["file_path"]) else: display_source_node(res_node, source_length=200) plot_images(retrieved_image)

节点 ID: 8c14be3e-345a-4764-9b64-dacff771bc04
相似性 0.8689195893277072
文本: === 旅游和会展 ===

文本: 旅游业是旧金山最重要的私营行业之一，占该市七分之一以上的工作岗位。该市经常被描绘为...

节点 ID: 22aa7d86-017f-433d-98dc-4007d9f67c17
相似性 0.8452524742723133
文本: === LGBT ===

文本: 旧金山长期以来拥有对 LGBT 友好的历史。它是美国第一个女同性恋权利组织 Daughters of Bilitis 的所在地；第一个公开同性恋身份竞选公职的人...

节点 ID: 3846a17a-79d8-415e-9bcf-76c818b27203
相似性 0.8329496262980858
文本: == 公园和休闲 ==

文本: 旧金山的几个公园和几乎所有的海滩都属于金门国家休闲区，这是全国访问量最大的地区之一...

In [ ]

test_query = "which company makes Tesla"
# generate  retrieval results
retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5)
retrieval_results = retriever.retrieve(test_query)

retrieved_image = []
for res_node in retrieval_results:
    if isinstance(res_node.node, ImageNode):
        retrieved_image.append(res_node.node.metadata["file_path"])
    else:
        display_source_node(res_node, source_length=200)

plot_images(retrieved_image)

test_query = "which company makes Tesla" # generate retrieval results retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5) retrieval_results = retriever.retrieve(test_query) retrieved_image = [] for res_node in retrieval_results: if isinstance(res_node.node, ImageNode): retrieved_image.append(res_node.node.metadata["file_path"]) else: display_source_node(res_node, source_length=200) plot_images(retrieved_image)

节点 ID: 214c61be-dad6-403c-b301-bc2320b87e7a
相似性 0.7808396168295813
文本: Tesla Model S 是一款电池电动全尺寸豪华轿车，采用掀背式车身风格，由 Tesla, Inc. 自 2012 年开始生产。Model S 采用电池供电的双电机全轮驱动布局，...

节点 ID: 15b737b4-90e3-443a-87aa-13a7d7e80b87
相似性 0.7807424063856144
文本: === P100D === P100D 在测功机上输出 439 kW (589 hp) 和 1,248 N⋅m (920 lbf⋅ft) 的扭矩。截至 2017 年 3 月，P100D 是世界上从 NHRA 起步到 6...

节点 ID: e134452b-3031-47b0-a20c-df4fe32f1bcf
相似性 0.7754107325086438
文本: === 召回 === 截至 2021 年 12 月，特斯拉共有七次 Model S 召回：2013 年 6 月 14 日，特斯拉召回了在 2013 年 5 月 10 日至 2013 年 6 月 8 日期间生产的 Model S 车辆，原因是方法不当...

In [ ]

test_query = "what is the main character in Batman"
# generate  retrieval results
retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5)
retrieval_results = retriever.retrieve(test_query)

retrieved_image = []
for res_node in retrieval_results:
    if isinstance(res_node.node, ImageNode):
        retrieved_image.append(res_node.node.metadata["file_path"])
    else:
        display_source_node(res_node, source_length=200)

plot_images(retrieved_image)

test_query = "what is the main character in Batman" # generate retrieval results retriever = index.as_retriever(similarity_top_k=3, image_similarity_top_k=5) retrieval_results = retriever.retrieve(test_query) retrieved_image = [] for res_node in retrieval_results: if isinstance(res_node.node, ImageNode): retrieved_image.append(res_node.node.metadata["file_path"]) else: display_source_node(res_node, source_length=200) plot_images(retrieved_image)

节点 ID: ff85b136-08c8-465d-96f5-a554c65067d8
相似性 0.8461934674061043
文本: ==== 性格 ==== 蝙蝠侠的主要性格特征可以概括为“财富、身体素质、推理能力和执着”。蝙蝠侠漫画书的细节和基调各不相同...

节点 ID: 55f5b842-6fd0-4e45-aef2-27f74f670e82
相似性 0.8229623965891602
文本: 蝙蝠侠是出现在 DC Comics 出版的美国漫画书中的超级英雄。这个角色由艺术家 Bob Kane 和编剧 Bill Finger 创作，首次亮相于该漫画书的第 27 期...

节点 ID: cb4755db-088e-46af-92b2-3a4a3649d9fe
相似性 0.8218281955244808
文本: === 敌人 ===

文本: 蝙蝠侠面对各种各样的敌人，从普通罪犯到古怪的超级反派。他们中的许多人反映了蝙蝠侠性格和发展的各个方面，通常带有悲剧性...