Redis 向量存储¶

在这个 Notebook 中，我们将快速演示如何使用 RedisVectorStore。

如果您在 colab 上打开此 Notebook，您可能需要安装 LlamaIndex 🦙。

输入 [ ]

%pip install -U llama-index llama-index-vector-stores-redis llama-index-embeddings-cohere llama-index-embeddings-openai

%pip install -U llama-index llama-index-vector-stores-redis llama-index-embeddings-cohere llama-index-embeddings-openai

输入 [ ]

import os
import getpass
import sys
import logging
import textwrap
import warnings

warnings.filterwarnings("ignore")

# Uncomment to see debug logs
logging.basicConfig(stream=sys.stdout, level=logging.INFO)

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.redis import RedisVectorStore

import os import getpass import sys import logging import textwrap import warnings warnings.filterwarnings("ignore") # Uncomment to see debug logs logging.basicConfig(stream=sys.stdout, level=logging.INFO) from llama_index.core import VectorStoreIndex, SimpleDirectoryReader from llama_index.vector_stores.redis import RedisVectorStore

启动 Redis¶

启动 Redis 最简单的方法是使用 Redis Stack docker 镜像，或者快速注册一个免费的 Redis Cloud 实例。

要按照本教程的每个步骤进行操作，请如下启动镜像

docker run --name redis-vecdb -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

这还将在 8001 端口启动 RedisInsight UI，您可以在 http://localhost:8001 查看。

设置 OpenAI¶

首先让我们添加 openai api 密钥。这将使我们能够访问 openai 获取嵌入并使用 chatgpt。

输入 [ ]

oai_api_key = getpass.getpass("OpenAI API Key:")
os.environ["OPENAI_API_KEY"] = oai_api_key

oai_api_key = getpass.getpass("OpenAI API Key:") os.environ["OPENAI_API_KEY"] = oai_api_key

下载数据

输入 [ ]

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

--2024-04-10 19:35:33--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8002::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8003::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: ‘data/paul_graham/paul_graham_essay.txt’

data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.03s   

2024-04-10 19:35:33 (2.15 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]

读取数据集¶

在这里，我们将使用一组 Paul Graham 的文章来提供文本以生成嵌入，存储在 RedisVectorStore 中，并通过查询找到我们 LLM 问答循环的上下文。

输入 [ ]

# load documents
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
print(
    "Document ID:",
    documents[0].id_,
    "Document Filename:",
    documents[0].metadata["file_name"],
)

# load documents documents = SimpleDirectoryReader("./data/paul_graham").load_data() print( "文档 ID:", documents[0].id_, "文档文件名:", documents[0].metadata["file_name"], )

Document ID: 7056f7ba-3513-4ef4-9792-2bd28040aaed Document Filename: paul_graham_essay.txt

初始化默认 Redis 向量存储¶

现在我们的文档已准备好，我们可以使用默认设置初始化 Redis 向量存储。这将使我们能够将向量存储在 Redis 中并创建索引以进行实时搜索。

输入 [ ]

from llama_index.core import StorageContext
from redis import Redis

# create a Redis client connection
redis_client = Redis.from_url("redis://localhost:6379")

# create the vector store wrapper
vector_store = RedisVectorStore(redis_client=redis_client, overwrite=True)

# load storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# build and load index from documents and storage context
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)
# index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

from llama_index.core import StorageContext from redis import Redis # 创建 Redis 客户端连接 redis_client = Redis.from_url("redis://localhost:6379") # 创建向量存储包装器 vector_store = RedisVectorStore(redis_client=redis_client, overwrite=True) # 加载存储上下文 storage_context = StorageContext.from_defaults(vector_store=vector_store) # 从文档和存储上下文构建并加载索引 index = VectorStoreIndex.from_documents( documents, storage_context=storage_context ) # index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

19:39:17 llama_index.vector_stores.redis.base INFO   Using default RedisVectorStore schema.
19:39:19 httpx INFO   HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
19:39:19 llama_index.vector_stores.redis.base INFO   Added 22 documents to index llama_index

查询默认向量存储¶

现在我们的数据已存储在索引中，我们可以针对索引提问。

该索引将使用数据作为 LLM 的知识库。as_query_engine() 的默认设置使用 OpenAI 嵌入和 GPT 作为语言模型。因此，除非您选择定制或本地语言模型，否则需要一个 OpenAI 密钥。

下面我们将测试针对索引的搜索，然后进行带有 LLM 的完整 RAG。

输入 [ ]

query_engine = index.as_query_engine()
retriever = index.as_retriever()

query_engine = index.as_query_engine() retriever = index.as_retriever()

输入 [ ]

result_nodes = retriever.retrieve("What did the author learn?")
for node in result_nodes:
    print(node)

result_nodes = retriever.retrieve("作者学到了什么？") for node in result_nodes: print(node)

19:39:22 httpx INFO   HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
19:39:22 llama_index.vector_stores.redis.base INFO   Querying index llama_index with filters *
19:39:22 llama_index.vector_stores.redis.base INFO   Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108']
Node ID: adb6b7ce-49bb-4961-8506-37082c02a389
Text: What I Worked On  February 2021  Before college the two main
things I worked on, outside of school, were writing and programming. I
didn't write essays. I wrote what beginning writers were supposed to
write then, and probably still are: short stories. My stories were
awful. They had hardly any plot, just characters with strong feelings,
which I ...
Score:  0.820

Node ID: e39be1fe-32d0-456e-b211-4efabd191108
Text: Except for a few officially anointed thinkers who went to the
right parties in New York, the only people allowed to publish essays
were specialists writing about their specialties. There were so many
essays that had never been written, because there had been no way to
publish them. Now they could be, and I was going to write them. [12]
I've wor...
Score:  0.819

输入 [ ]

response = query_engine.query("What did the author learn?")
print(textwrap.fill(str(response), 100))

response = query_engine.query("作者学到了什么？") print(textwrap.fill(str(response), 100))

19:39:25 httpx INFO   HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
19:39:25 llama_index.vector_stores.redis.base INFO   Querying index llama_index with filters *
19:39:25 llama_index.vector_stores.redis.base INFO   Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108']
19:39:27 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
The author learned that working on things that weren't prestigious often led to valuable discoveries
and indicated the right kind of motives. Despite the lack of initial prestige, pursuing such work
could be a sign of genuine potential and appropriate motivations, steering clear of the common
pitfall of being driven solely by the desire to impress others.

输入 [ ]

result_nodes = retriever.retrieve("What was a hard moment for the author?")
for node in result_nodes:
    print(node)

result_nodes = retriever.retrieve("作者的艰难时刻是什么？") for node in result_nodes: print(node)

19:39:27 httpx INFO   HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
19:39:27 llama_index.vector_stores.redis.base INFO   Querying index llama_index with filters *
19:39:27 llama_index.vector_stores.redis.base INFO   Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108']
Node ID: adb6b7ce-49bb-4961-8506-37082c02a389
Text: What I Worked On  February 2021  Before college the two main
things I worked on, outside of school, were writing and programming. I
didn't write essays. I wrote what beginning writers were supposed to
write then, and probably still are: short stories. My stories were
awful. They had hardly any plot, just characters with strong feelings,
which I ...
Score:  0.802

Node ID: e39be1fe-32d0-456e-b211-4efabd191108
Text: Except for a few officially anointed thinkers who went to the
right parties in New York, the only people allowed to publish essays
were specialists writing about their specialties. There were so many
essays that had never been written, because there had been no way to
publish them. Now they could be, and I was going to write them. [12]
I've wor...
Score:  0.799

输入 [ ]

response = query_engine.query("What was a hard moment for the author?")
print(textwrap.fill(str(response), 100))

response = query_engine.query("作者的艰难时刻是什么？") print(textwrap.fill(str(response), 100))

19:39:29 httpx INFO   HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
19:39:29 llama_index.vector_stores.redis.base INFO   Querying index llama_index with filters *
19:39:29 llama_index.vector_stores.redis.base INFO   Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108']
19:39:31 httpx INFO   HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
A hard moment for the author was when one of his programs on the IBM 1401 mainframe didn't
terminate, leading to a technical error and an uncomfortable situation with the data center manager.

输入 [ ]

index.vector_store.delete_index()

index.vector_store.delete_index()

19:39:34 llama_index.vector_stores.redis.base INFO   Deleting index llama_index

使用自定义索引模式¶

在大多数使用案例中，您需要能够自定义底层索引配置和规范。例如，这对于定义您希望启用的特定元数据过滤器非常方便。

使用 Redis，这就像定义一个索引模式对象 (从文件或字典) 并将其传递给向量存储客户端包装器一样简单。

在此示例中，我们将

1. 将嵌入模型切换到 Cohere
2. 为文档添加一个额外的 metadata 字段用于 updated_at 时间戳
3. 索引现有的 file_name metadata 字段

输入 [ ]

from llama_index.core.settings import Settings
from llama_index.embeddings.cohere import CohereEmbedding

# set up Cohere Key
co_api_key = getpass.getpass("Cohere API Key:")
os.environ["CO_API_KEY"] = co_api_key

# set llamaindex to use Cohere embeddings
Settings.embed_model = CohereEmbedding()

from llama_index.core.settings import Settings from llama_index.embeddings.cohere import CohereEmbedding # 设置 Cohere 密钥 co_api_key = getpass.getpass("Cohere API Key:") os.environ["CO_API_KEY"] = co_api_key # 设置 llamaindex 使用 Cohere 嵌入 Settings.embed_model = CohereEmbedding()

输入 [ ]

from redisvl.schema import IndexSchema


custom_schema = IndexSchema.from_dict(
    {
        # customize basic index specs
        "index": {
            "name": "paul_graham",
            "prefix": "essay",
            "key_separator": ":",
        },
        # customize fields that are indexed
        "fields": [
            # required fields for llamaindex
            {"type": "tag", "name": "id"},
            {"type": "tag", "name": "doc_id"},
            {"type": "text", "name": "text"},
            # custom metadata fields
            {"type": "numeric", "name": "updated_at"},
            {"type": "tag", "name": "file_name"},
            # custom vector field definition for cohere embeddings
            {
                "type": "vector",
                "name": "vector",
                "attrs": {
                    "dims": 1024,
                    "algorithm": "hnsw",
                    "distance_metric": "cosine",
                },
            },
        ],
    }
)

from redisvl.schema import IndexSchema custom_schema = IndexSchema.from_dict( { # 自定义基本索引规范 "index": { "name": "paul_graham", "prefix": "essay", "key_separator": ":", }, # 自定义被索引的字段 "fields": [ # llamaindex 所需字段 {"type": "tag", "name": "id"}, {"type": "tag", "name": "doc_id"}, {"type": "text", "name": "text"}, # 自定义元数据字段 {"type": "numeric", "name": "updated_at"}, {"type": "tag", "name": "file_name"}, # 用于 cohere 嵌入的自定义向量字段定义 { "type": "vector", "name": "vector", "attrs": { "dims": 1024, "algorithm": "hnsw", "distance_metric": "cosine", }, }, ], } )

输入 [ ]

custom_schema.index

custom_schema.index

输出 [ ]

IndexInfo(name='paul_graham', prefix='essay', key_separator=':', storage_type=<StorageType.HASH: 'hash'>)

输入 [ ]

custom_schema.fields

custom_schema.fields

输出 [ ]

{'id': TagField(name='id', type='tag', path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),
 'doc_id': TagField(name='doc_id', type='tag', path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),
 'text': TextField(name='text', type='text', path=None, attrs=TextFieldAttributes(sortable=False, weight=1, no_stem=False, withsuffixtrie=False, phonetic_matcher=None)),
 'updated_at': NumericField(name='updated_at', type='numeric', path=None, attrs=NumericFieldAttributes(sortable=False)),
 'file_name': TagField(name='file_name', type='tag', path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),
 'vector': HNSWVectorField(name='vector', type='vector', path=None, attrs=HNSWVectorFieldAttributes(dims=1024, algorithm=<VectorIndexAlgorithm.HNSW: 'HNSW'>, datatype=<VectorDataType.FLOAT32: 'FLOAT32'>, distance_metric=<VectorDistanceMetric.COSINE: 'COSINE'>, initial_cap=None, m=16, ef_construction=200, ef_runtime=10, epsilon=0.01))}

了解更多关于使用 redis 的模式和索引设计。

输入 [ ]

from datetime import datetime


def date_to_timestamp(date_string: str) -> int:
    date_format: str = "%Y-%m-%d"
    return int(datetime.strptime(date_string, date_format).timestamp())


# iterate through documents and add new field
for document in documents:
    document.metadata["updated_at"] = date_to_timestamp(
        document.metadata["last_modified_date"]
    )

from datetime import datetime def date_to_timestamp(date_string: str) -> int: date_format: str = "%Y-%m-%d" return int(datetime.strptime(date_string, date_format).timestamp()) # 遍历文档并添加新字段 for document in documents: document.metadata["updated_at"] = date_to_timestamp( document.metadata["last_modified_date"] )

输入 [ ]

vector_store = RedisVectorStore(
    schema=custom_schema,  # provide customized schema
    redis_client=redis_client,
    overwrite=True,
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

# build and load index from documents and storage context
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

vector_store = RedisVectorStore( schema=custom_schema, # 提供自定义模式 redis_client=redis_client, overwrite=True, ) storage_context = StorageContext.from_defaults(vector_store=vector_store) # 从文档和存储上下文构建并加载索引 index = VectorStoreIndex.from_documents( documents, storage_context=storage_context )

19:40:05 httpx INFO   HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK"
19:40:06 httpx INFO   HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK"
19:40:06 httpx INFO   HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK"
19:40:06 llama_index.vector_stores.redis.base INFO   Added 22 documents to index paul_graham

查询向量存储并基于元数据进行过滤¶

现在我们已经在 Redis 中索引了额外的元数据，让我们尝试一些带过滤器的查询。

输入 [ ]

from llama_index.core.vector_stores import (
    MetadataFilters,
    MetadataFilter,
    ExactMatchFilter,
)

retriever = index.as_retriever(
    similarity_top_k=3,
    filters=MetadataFilters(
        filters=[
            ExactMatchFilter(key="file_name", value="paul_graham_essay.txt"),
            MetadataFilter(
                key="updated_at",
                value=date_to_timestamp("2023-01-01"),
                operator=">=",
            ),
            MetadataFilter(
                key="text",
                value="learn",
                operator="text_match",
            ),
        ],
        condition="and",
    ),
)

from llama_index.core.vector_stores import ( MetadataFilters, MetadataFilter, ExactMatchFilter, ) retriever = index.as_retriever( similarity_top_k=3, filters=MetadataFilters( filters=[ ExactMatchFilter(key="file_name", value="paul_graham_essay.txt"), MetadataFilter( key="updated_at", value=date_to_timestamp("2023-01-01"), operator=">=", ), MetadataFilter( key="text", value="learn", operator="text_match", ), ], condition="and", ), )

输入 [ ]

result_nodes = retriever.retrieve("What did the author learn?")

for node in result_nodes:
    print(node)

result_nodes = retriever.retrieve("作者学到了什么？") for node in result_nodes: print(node)

19:40:22 httpx INFO   HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK"

19:40:22 llama_index.vector_stores.redis.base INFO   Querying index paul_graham with filters ((@file_name:{paul_graham_essay\.txt} @updated_at:[1672549200 +inf]) @text:(learn))
19:40:22 llama_index.vector_stores.redis.base INFO   Found 3 results for query with id ['essay:0df3b734-ecdb-438e-8c90-f21a8c80f552', 'essay:01108c0d-140b-4dcc-b581-c38b7df9251e', 'essay:ced36463-ac36-46b0-b2d7-935c1b38b781']
Node ID: 0df3b734-ecdb-438e-8c90-f21a8c80f552
Text: All that seemed left for philosophy were edge cases that people
in other fields felt could safely be ignored.  I couldn't have put
this into words when I was 18. All I knew at the time was that I kept
taking philosophy courses and they kept being boring. So I decided to
switch to AI.  AI was in the air in the mid 1980s, but there were two
things...
Score:  0.410

Node ID: 01108c0d-140b-4dcc-b581-c38b7df9251e
Text: It was not, in fact, simply a matter of teaching SHRDLU more
words. That whole way of doing AI, with explicit data structures
representing concepts, was not going to work. Its brokenness did, as
so often happens, generate a lot of opportunities to write papers
about various band-aids that could be applied to it, but it was never
going to get us ...
Score:  0.390

Node ID: ced36463-ac36-46b0-b2d7-935c1b38b781
Text: Grad students could take classes in any department, and my
advisor, Tom Cheatham, was very easy going. If he even knew about the
strange classes I was taking, he never said anything.  So now I was in
a PhD program in computer science, yet planning to be an artist, yet
also genuinely in love with Lisp hacking and working away at On Lisp.
In other...
Score:  0.389

从 Redis 中的现有索引恢复¶

从索引恢复需要一个 Redis 连接客户端 (或 URL)，overwrite=False，并传入之前使用的相同模式对象。(这可以使用 .to_yaml() 方便地卸载到 YAML 文件中)

输入 [ ]

custom_schema.to_yaml("paul_graham.yaml")

custom_schema.to_yaml("paul_graham.yaml")

输入 [ ]

vector_store = RedisVectorStore(
    schema=IndexSchema.from_yaml("paul_graham.yaml"),
    redis_client=redis_client,
)
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

vector_store = RedisVectorStore( schema=IndexSchema.from_yaml("paul_graham.yaml"), redis_client=redis_client, ) index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

19:40:28 redisvl.index.index INFO   Index already exists, not overwriting.

在不久的将来 -- 我们将实现一个便捷方法，仅使用索引名称进行加载

RedisVectorStore.from_existing_index(index_name="paul_graham", redis_client=redis_client)

完全删除文档或索引¶

有时可能需要删除文档或整个索引。这可以通过使用 delete 和 delete_index 方法完成。

输入 [ ]

document_id = documents[0].doc_id
document_id

文档 ID = documents[0].doc_id document_id

输出 [ ]

'7056f7ba-3513-4ef4-9792-2bd28040aaed'

输入 [ ]

print("Number of documents before deleting", redis_client.dbsize())
vector_store.delete(document_id)
print("Number of documents after deleting", redis_client.dbsize())

print("删除前的文档数量", redis_client.dbsize()) vector_store.delete(document_id) print("删除后的文档数量", redis_client.dbsize())

Number of documents before deleting 22
19:40:32 llama_index.vector_stores.redis.base INFO   Deleted 22 documents from index paul_graham
Number of documents after deleting 0

然而，Redis 索引仍然存在 (没有关联的文档)，以便进行持续的 upsert 操作。

输入 [ ]

vector_store.index_exists()

vector_store.index_exists()

输出 [ ]

True

输入 [ ]

# now lets delete the index entirely
# this will delete all the documents and the index
vector_store.delete_index()

# 现在我们完全删除索引 # 这将删除所有文档和索引 vector_store.delete_index()

19:40:37 llama_index.vector_stores.redis.base INFO   Deleting index paul_graham

输入 [ ]

print("Number of documents after deleting", redis_client.dbsize())

print("删除后的文档数量", redis_client.dbsize())

Number of documents after deleting 0

故障排除¶

如果您获得空查询结果，有几个问题需要检查

模式¶

与其他向量存储不同，Redis 要求用户显式定义索引的模式。原因如下：

1. Redis 被用于多种使用案例，包括实时向量搜索，但也用于标准文档存储/检索、缓存、消息、发布/订阅、会话管理等。并非所有记录上的属性都需要为搜索而索引。这部分是为了效率，部分是为了尽量减少用户失误。
2. 使用 Redis 和 LlamaIndex 时，所有索引模式至少必须包含以下字段：id, doc_id, text, 和 vector。

使用默认模式 (假定使用 OpenAI 嵌入) 或自定义模式 (参见上文) 实例化您的 RedisVectorStore。

前缀问题¶

Redis 要求所有记录都有一个键前缀，将键空间分割成“分区”，以供可能不同的应用、使用案例和客户端使用。

确保所选的前缀作为索引模式的一部分，在您的代码中保持一致 (与特定索引绑定)。

要查看您的索引是使用哪个前缀创建的，您可以在 Redis CLI 中运行 FT.INFO <您的索引名称>，然后在 index_definition => prefixes 下查找。

数据 vs 索引¶

Redis 将数据集中的记录和索引视为不同的实体。这使您在执行更新、upsert 和索引模式迁移时具有更大的灵活性。

如果您有一个现有索引并想确保其被删除，您可以在 Redis CLI 中运行 FT.DROPINDEX <您的索引名称>。请注意，除非您传入 DD，否则这不会删除您的实际数据。

使用元数据时查询结果为空¶

如果您在索引已经创建后向其添加元数据，然后尝试基于该元数据进行查询，您的查询将返回空结果。

Redis 只在索引创建时索引字段 (类似于上面它索引前缀的方式)。