使用向量存储

LlamaIndex 提供与向量存储/向量数据库的多个集成点

1. LlamaIndex 可以将向量存储本身用作索引。像任何其他索引一样，此索引可以存储文档并用于回答查询。
2. LlamaIndex 可以从向量存储加载数据，类似于任何其他数据连接器。然后可以在 LlamaIndex 数据结构中使用此数据。

使用向量存储作为索引

LlamaIndex 还支持不同的向量存储作为 VectorStoreIndex 的存储后端。

• 阿里云 OpenSearch (AlibabaCloudOpenSearchStore)。 快速入门。
• Amazon Neptune - Neptune Analytics (NeptuneAnalyticsVectorStore)。 在 Neptune Analytics 中处理向量相似度。
• 通过 CQL 连接 Apache Cassandra® 和 Astra DB (CassandraVectorStore)。 安装 快速入门
• Astra DB (AstraDBVectorStore)。 快速入门。
• AWS Document DB (AWSDocDbVectorStore)。 快速入门。
• Azure AI Search (AzureAISearchVectorStore)。 快速入门
• Chroma (ChromaVectorStore) 安装
• ClickHouse (ClickHouseVectorStore) 安装
• Couchbase (CouchbaseVectorStore) 安装
• DashVector (DashVectorStore)。 安装。
• DeepLake (DeepLakeVectorStore) 安装
• DocArray (DocArrayHnswVectorStore, DocArrayInMemoryVectorStore)。 安装/Python 客户端。
• Elasticsearch (ElasticsearchStore) 安装
• Epsilla (EpsillaVectorStore) 安装/快速入门
• Faiss (FaissVectorStore)。 安装。
• Google AlloyDB for PostgreSQL (AlloyDBVectorStore)。 快速入门。
• Google Cloud SQL for PostgreSQL (PostgresVectorStore)。 快速入门
• Hnswlib (HnswlibVectorStore)。 安装。
• txtai (TxtaiVectorStore)。 安装。
• Jaguar (JaguarVectorStore)。 安装。
• Lantern (LanternVectorStore)。 快速入门。
• MariaDB (MariaDBVectorStore)。 MariaDB Vector 概述
• Milvus (MilvusVectorStore)。 安装
• MongoDB Atlas (MongoDBAtlasVectorSearch)。 安装/快速入门。
• MyScale (MyScaleVectorStore)。 快速入门。 安装/Python 客户端。
• Neo4j (Neo4jVectorIndex)。 安装。
• OceanBase (OceanBaseVectorStore)。 OceanBase 概述。 快速入门。 Python 客户端
• Opensearch (OpensearchVectorStore) 将 Opensearch 用作向量数据库。 快速入门
• Pinecone (PineconeVectorStore)。 安装/快速入门。
• Qdrant (QdrantVectorStore) 安装 Python 客户端
• LanceDB (LanceDBVectorStore) 安装/快速入门
• Redis (RedisVectorStore)。 安装。
• Relyt (RelytVectorStore)。 快速入门。
• Supabase (SupabaseVectorStore)。 快速入门。
• Tablestore (Tablestore)。 Tablestore 概述。 快速入门。 Python 客户端。
• TiDB (TiDBVectorStore)。 快速入门。 安装。 Python 客户端。
• TimeScale (TimescaleVectorStore)。 安装。
• Upstash (UpstashVectorStore)。 快速入门
• Vertex AI Vector Search (VertexAIVectorStore)。 快速入门
• Weaviate (WeaviateVectorStore)。 安装。 Python 客户端。
• WordLift (WordliftVectorStore)。 快速入门。 Python 客户端。
• Zep (ZepVectorStore)。 安装。 Python 客户端。
• Zilliz (MilvusVectorStore)。 快速入门

详细的 API 参考可以在此处找到。

类似于 LlamaIndex 中的任何其他索引 (树状、关键词表、列表)，VectorStoreIndex 可以在任何文档集合上构建。我们在索引内部使用向量存储来存储输入文本块的嵌入。

构建后，该索引可用于查询。

默认向量存储索引构建/查询

默认情况下，VectorStoreIndex 使用作为默认存储上下文一部分初始化的内存中 SimpleVectorStore。

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Load documents and build index
documents = SimpleDirectoryReader("../paul_graham_essay/data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query index
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")

自定义向量存储索引构建/查询

我们可以按如下方式查询自定义向量存储

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
)
from llama_index.vector_stores.deeplake import DeepLakeVectorStore

# construct vector store and customize storage context
storage_context = StorageContext.from_defaults(
    vector_store=DeepLakeVectorStore(dataset_path="<dataset_path>")
)

# Load documents and build index
documents = SimpleDirectoryReader("../paul_graham_essay/data").load_data()
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# Query index
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")

下面展示了如何构建我们支持的各种向量存储的更多示例。

阿里云 OpenSearch

from llama_index.vector_stores.alibabacloud_opensearch import (
    AlibabaCloudOpenSearchStore,
    AlibabaCloudOpenSearchConfig,
)

config = AlibabaCloudOpenSearchConfig(
    endpoint="***",
    instance_id="***",
    username="your_username",
    password="your_password",
    table_name="llama",
)

vector_store = AlibabaCloudOpenSearchStore(config)

Google AlloyDB for PostgreSQL

pip install llama-index
pip install llama-index-alloydb-pg
pip install llama-index-llms-vertex
gcloud services enable aiplatform.googleapis.com

from llama_index_alloydb_pg import AlloyDBEngine, AlloyDBVectorStore
from llama_index.core import Settings
from llama_index.embeddings.vertex import VertexTextEmbedding
from llama_index.llms.vertex import Vertex
import google.auth

# Replace with your own AlloyDB info
engine = AlloyDBEngine.from_instance(
    project_id=PROJECT_ID,
    region=REGION,
    cluster=CLUSTER,
    instance=INSTANCE,
    database=DATABASE,
    user=USER,
    password=PASSWORD,
)

engine.init_vector_store_table(
    table_name=TABLE_NAME,
    vector_size=768,  # Vector size for VertexAI model(textembedding-gecko@latest)
)

vector_store = AlloyDBVectorStore.create_sync(
    engine=engine,
    table_name=TABLE_NAME,
)

Amazon Neptune - Neptune Analytics

from llama_index.vector_stores.neptune import NeptuneAnalyticsVectorStore

graph_identifier = ""
embed_dim = 1536

neptune_vector_store = NeptuneAnalyticsVectorStore(
    graph_identifier=graph_identifier, embedding_dimension=1536
)

Apache Cassandra®

from llama_index.vector_stores.cassandra import CassandraVectorStore
import cassio

# To use an Astra DB cloud instance through CQL:
cassio.init(database_id="1234abcd-...", token="AstraCS:...")

# For a Cassandra cluster:
from cassandra.cluster import Cluster

cluster = Cluster(["127.0.0.1"])
cassio.init(session=cluster.connect(), keyspace="my_keyspace")

# After the above `cassio.init(...)`, create a vector store:
vector_store = CassandraVectorStore(
    table="cass_v_table", embedding_dimension=1536
)

Astra DB

from llama_index.vector_stores.astra_db import AstraDBVectorStore

astra_db_store = AstraDBVectorStore(
    token="AstraCS:xY3b...",  # Your Astra DB token
    api_endpoint="https://012...abc-us-east1.apps.astra.datastax.com",  # Your Astra DB API endpoint
    collection_name="astra_v_table",  # Table name of your choice
    embedding_dimension=1536,  # Embedding dimension of the embeddings model used
)

Azure Cognitive Search

from azure.core.credentials import AzureKeyCredential
from llama_index.vector_stores.azureaisearch import AzureAISearchVectorStore

search_service_api_key = "YOUR-AZURE-SEARCH-SERVICE-ADMIN-KEY"
search_service_endpoint = "YOUR-AZURE-SEARCH-SERVICE-ENDPOINT"
search_service_api_version = "2023-11-01"
credential = AzureKeyCredential(search_service_api_key)

# Index name to use
index_name = "llamaindex-vector-demo"

client = SearchIndexClient(
    endpoint=search_service_endpoint,
    credential=credential,
)

vector_store = AzureAISearchVectorStore(
    search_or_index_client=client,
    index_name=index_name,
    embedding_dimensionality=1536,
)

Chroma

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore

# Creating a Chroma client
# EphemeralClient operates purely in-memory, PersistentClient will also save to disk
chroma_client = chromadb.EphemeralClient()
chroma_collection = chroma_client.create_collection("quickstart")

# construct vector store
vector_store = ChromaVectorStore(
    chroma_collection=chroma_collection,
)

ClickHouse

import clickhouse_connect
from llama_index.vector_stores import ClickHouseVectorStore

# Creating a ClickHouse client
client = clickhouse_connect.get_client(
    host="YOUR_CLUSTER_HOST",
    port=8123,
    username="YOUR_USERNAME",
    password="YOUR_CLUSTER_PASSWORD",
)

# construct vector store
vector_store = ClickHouseVectorStore(clickhouse_client=client)

Couchdb

from datetime import timedelta

from couchbase.auth import PasswordAuthenticator
from couchbase.cluster import Cluster
from couchbase.options import ClusterOptions

# Create a Couchbase Cluster object
auth = PasswordAuthenticator("DATABASE_USERNAME", "DATABASE_PASSWORD")
options = ClusterOptions(auth)
cluster = Cluster("CLUSTER_CONNECTION_STRING", options)

# Wait until the cluster is ready for use.
cluster.wait_until_ready(timedelta(seconds=5))

# Create the Vector Store
vector_store = CouchbaseVectorStore(
    cluster=cluster,
    bucket_name="BUCKET_NAME",
    scope_name="SCOPE_NAME",
    collection_name="COLLECTION_NAME",
    index_name="SEARCH_INDEX_NAME",
)

DashVector

import dashvector
from llama_index.vector_stores.dashvector import DashVectorStore

# init dashvector client
client = dashvector.Client(
    api_key="your-dashvector-api-key",
    endpoint="your-dashvector-cluster-endpoint",
)

# creating a DashVector collection
client.create("quickstart", dimension=1536)
collection = client.get("quickstart")

# construct vector store
vector_store = DashVectorStore(collection)

DeepLake

import os
import getpath
from llama_index.vector_stores.deeplake import DeepLakeVectorStore

os.environ["OPENAI_API_KEY"] = getpath.getpath("OPENAI_API_KEY: ")
os.environ["ACTIVELOOP_TOKEN"] = getpath.getpath("ACTIVELOOP_TOKEN: ")
dataset_path = "hub://adilkhan/paul_graham_essay"

# construct vector store
vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True)

DocArray

from llama_index.vector_stores.docarray import (
    DocArrayHnswVectorStore,
    DocArrayInMemoryVectorStore,
)

# construct vector store
vector_store = DocArrayHnswVectorStore(work_dir="hnsw_index")

# alternatively, construct the in-memory vector store
vector_store = DocArrayInMemoryVectorStore()

Elasticsearch

首先，您可以本地启动 Elasticsearch 或在 Elastic Cloud 上启动。

要使用 docker 本地启动 Elasticsearch，请运行以下命令

docker run -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -e "xpack.security.http.ssl.enabled=false" \
  -e "xpack.license.self_generated.type=trial" \
  docker.elastic.co/elasticsearch/elasticsearch:8.9.0

然后连接并使用 Elasticsearch 作为 LlamaIndex 的向量数据库

from llama_index.vector_stores.elasticsearch import ElasticsearchStore

vector_store = ElasticsearchStore(
    index_name="llm-project",
    es_url="http://localhost:9200",
    # Cloud connection options:
    # es_cloud_id="<cloud_id>",
    # es_user="elastic",
    # es_password="<password>",
)

这可以与 VectorStoreIndex 一起使用，提供用于检索、查询、删除、持久化索引等的查询接口。

Epsilla

from pyepsilla import vectordb
from llama_index.vector_stores.epsilla import EpsillaVectorStore

# Creating an Epsilla client
epsilla_client = vectordb.Client()

# Construct vector store
vector_store = EpsillaVectorStore(client=epsilla_client)

注意：EpsillaVectorStore 依赖于 pyepsilla 库和运行中的 Epsilla 向量数据库。如果尚未安装，请使用 pip/pip3 install pyepsilla。可以通过 docker 镜像找到运行中的 Epsilla 向量数据库。有关完整说明，请参阅以下文档：https://epsilla-inc.gitbook.io/epsilladb/quick-start

Feedly rss

import faiss
from llama_index.vector_stores.faiss import FaissVectorStore

# create faiss index
d = 1536
faiss_index = faiss.IndexFlatL2(d)

# construct vector store
vector_store = FaissVectorStore(faiss_index)

# if update/delete functionality is needed you can leverage the FaissMapVectorStore

d = 1536
faiss_index = faiss.IndexFlatL2(d)
id_map_index = faiss.IndexIDMap2(faiss_index)
vector_store = FaissMapVectorStore(id_map_index)

...

# NOTE: since faiss index is in-memory, we need to explicitly call
#       vector_store.persist() or storage_context.persist() to save it to disk.
#       persist() takes in optional arg persist_path. If none give, will use default paths.
storage_context.persist()

Google Cloud SQL for PostgreSQL

pip install llama-index
pip install llama-index-cloud-sql-pg
pip install llama-index-llms-vertex
gcloud services enable aiplatform.googleapis.com

from llama_index_cloud_sql_pg import PostgresEngine, PostgresVectorStore
from llama_index.core import Settings
from llama_index.embeddings.vertex import VertexTextEmbedding
from llama_index.llms.vertex import Vertex
import google.auth

# Replace with your own Cloud SQL info
engine = PostgresEngine.from_instance(
    project_id=PROJECT_ID,
    region=REGION,
    instance=INSTANCE,
    database=DATABASE,
    user=USER,
    password=PASSWORD,
)

engine.init_vector_store_table(
    table_name=TABLE_NAME,
    vector_size=768,  # Vector size for VertexAI model(textembedding-gecko@latest)
)

vector_store = PostgresVectorStore.create_sync(
    engine=engine,
    table_name=TABLE_NAME,
)

txtai

import txtai
from llama_index.vector_stores.txtai import TxtaiVectorStore

# create txtai index
txtai_index = txtai.ann.ANNFactory.create(
    {"backend": "numpy", "dimension": 512}
)

# construct vector store
vector_store = TxtaiVectorStore(txtai_index)

Jira

from llama_index.core.schema import TextNode
from llama_index.core.vector_stores import VectorStoreQuery
from jaguardb_http_client.JaguarHttpClient import JaguarHttpClient
from llama_index.vector_stores.jaguar import JaguarVectorStore


# construct vector store client
url = "http://127.0.0.1:8080/fwww/"
pod = "vdb"
store = "llamaindex_rag_store"
vector_index = "v"
vector_type = "cosine_fraction_float"
vector_dimension = 3

# require JAGUAR_API_KEY environment variable or file $HOME/.jagrc to hold the
# jaguar API key to connect to jaguar store server
vector_store = JaguarVectorStore(
    pod, store, vector_index, vector_type, vector_dimension, url
)

# login to jaguar server for security authentication
vector_store.login()

# create a vector store on the back-end server
metadata_fields = "author char(32), category char(16)"
text_size = 1024
vector_store.create(metadata_fields, text_size)

# store some text
node = TextNode(
    text="Return of King Lear",
    metadata={"author": "William", "category": "Tragedy"},
    embedding=[0.9, 0.1, 0.4],
)
vector_store.add(nodes=[node], use_node_metadata=True)

# make a query
qembedding = [0.4, 0.2, 0.8]
vsquery = VectorStoreQuery(query_embedding=qembedding, similarity_top_k=1)
query_result = vector_store.query(vsquery)

# make a query with metadata filter (where condition)
qembedding = [0.6, 0.1, 0.4]
vsquery = VectorStoreQuery(query_embedding=qembedding, similarity_top_k=3)
where = "author='Eve' or (author='Adam' and category='History')"
query_result = vector_store.query(vsquery, where=where)

# make a query ignoring old data (with time cutoff)
qembedding = [0.3, 0.3, 0.8]
vsquery = VectorStoreQuery(query_embedding=qembedding, similarity_top_k=3)
args = "day_cutoff=180"  # only search recent 180 days data
query_result = vector_store.query(vsquery, args=args)

# check if a vector is anomalous
text = ("Gone With The Wind",)
embed_of_text = [0.7, 0.1, 0.2]
node = TextNode(text=text, embedding=embed_of_text)
true_or_false = vector_store.is_anomalous(node)

# llama_index RAG application
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import StorageContext
from llama_index.core import VectorStoreIndex

question = "What did the author do growing up?"

storage_context = StorageContext.from_defaults(vector_store=vector_store)
embed_model = OpenAIEmbedding()
embed_of_question = [0.7, 0.1, 0.2]

db_documents = vector_store.load_documents(embed_of_question, 10)
index = VectorStoreIndex.from_documents(
    db_documents,
    embed_model=embed_model,
    storage_context=storage_context,
)

query_engine = index.as_query_engine()
print(f"Question: {question}")
response = query_engine.query(question)
print(f"Answer: {str(response)}")

# logout to clean up resources
vector_store.logout()

注意：客户端(需要 jaguardb-http-client) <--> Http Gateway <--> JaguarDB Server 客户端需要运行："pip install -U jaguardb-http-client"

MariaDB

from llama_index.vector_stores.mariadb import MariaDBVectorStore

vector_store = MariaDBVectorStore.from_params(
    host="localhost",
    port=3306,
    user="llamaindex",
    password="password",
    database="vectordb",
    table_name="llama_index_vectorstore",
    embed_dim=1536,  # OpenAI embedding dimension
)

Minio

• Milvus Index 提供了存储文档及其嵌入的功能。

import pymilvus
from llama_index.vector_stores.milvus import MilvusVectorStore

# construct vector store
vector_store = MilvusVectorStore(
    uri="https://localhost:19530", overwrite="True"
)

注意：MilvusVectorStore 依赖于 pymilvus 库。如果尚未安装，请使用 pip install pymilvus。如果您在构建 grpcio 的 wheel 时遇到问题，请检查您是否正在使用 python 3.11 (存在已知问题：https://github.com/milvus-io/pymilvus/issues/1308) 并尝试降级。

MongoDBAtlas

# Provide URI to constructor, or use environment variable
import pymongo
from llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearch
from llama_index.core import VectorStoreIndex
from llama_index.core import StorageContext
from llama_index.core import SimpleDirectoryReader

# mongo_uri = os.environ["MONGO_URI"]
mongo_uri = (
    "mongodb+srv://<username>:<password>@<host>?retryWrites=true&w=majority"
)
mongodb_client = pymongo.MongoClient(mongo_uri)

# construct store
store = MongoDBAtlasVectorSearch(mongodb_client)
storage_context = StorageContext.from_defaults(vector_store=store)
uber_docs = SimpleDirectoryReader(
    input_files=["../data/10k/uber_2021.pdf"]
).load_data()

# construct index
index = VectorStoreIndex.from_documents(
    uber_docs, storage_context=storage_context
)

MyScale

import clickhouse_connect
from llama_index.vector_stores.myscale import MyScaleVectorStore

# Creating a MyScale client
client = clickhouse_connect.get_client(
    host="YOUR_CLUSTER_HOST",
    port=8443,
    username="YOUR_USERNAME",
    password="YOUR_CLUSTER_PASSWORD",
)


# construct vector store
vector_store = MyScaleVectorStore(myscale_client=client)

Neo4j

• Neo4j 存储文本、元数据和嵌入，并且可以定制以元数据的形式返回图数据。

from llama_index.vector_stores.neo4jvector import Neo4jVectorStore

# construct vector store
neo4j_vector = Neo4jVectorStore(
    username="neo4j",
    password="pleaseletmein",
    url="bolt://localhost:7687",
    embed_dim=1536,
)

Pinecone

import pinecone
from llama_index.vector_stores.pinecone import PineconeVectorStore

# Creating a Pinecone index
api_key = "api_key"
pinecone.init(api_key=api_key, environment="us-west1-gcp")
pinecone.create_index(
    "quickstart", dimension=1536, metric="euclidean", pod_type="p1"
)
index = pinecone.Index("quickstart")

# construct vector store
vector_store = PineconeVectorStore(pinecone_index=index)

Quip

import qdrant_client
from llama_index.vector_stores.qdrant import QdrantVectorStore

# Creating a Qdrant vector store
client = qdrant_client.QdrantClient(
    host="<qdrant-host>", api_key="<qdrant-api-key>", https=True
)
collection_name = "paul_graham"

# construct vector store
vector_store = QdrantVectorStore(
    client=client,
    collection_name=collection_name,
)

Redis

首先，启动 Redis-Stack (或从 Redis 提供商获取 url)

docker run --name redis-vecdb -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

然后连接并使用 Redis 作为 LlamaIndex 的向量数据库

from llama_index.vector_stores.redis import RedisVectorStore

vector_store = RedisVectorStore(
    index_name="llm-project",
    redis_url="redis://localhost:6379",
    overwrite=True,
)

这可以与 VectorStoreIndex 一起使用，提供用于检索、查询、删除、持久化索引等的查询接口。

SingleStore

from llama_index.vector_stores.singlestoredb import SingleStoreVectorStore
import os

# can set the singlestore db url in env
# or pass it in as an argument to the SingleStoreVectorStore constructor
os.environ["SINGLESTOREDB_URL"] = "PLACEHOLDER URL"
vector_store = SingleStoreVectorStore(
    table_name="embeddings",
    content_field="content",
    metadata_field="metadata",
    vector_field="vector",
    timeout=30,
)

Tablestore

import tablestore
from llama_index.vector_stores.tablestore import TablestoreVectorStore

# create a vector store that does not support filtering non-vector fields
simple_vector_store = TablestoreVectorStore(
    endpoint="<end_point>",
    instance_name="<instance_name>",
    access_key_id="<access_key_id>",
    access_key_secret="<access_key_secret>",
    vector_dimension=512,
)

# create a vector store that support filtering non-vector fields
vector_store_with_meta_data = TablestoreVectorStore(
    endpoint="<end_point>",
    instance_name="<instance_name>",
    access_key_id="<access_key_id>",
    access_key_secret="<access_key_secret>",
    vector_dimension=512,
    # optional: custom metadata mapping is used to filter non-vector fields.
    metadata_mappings=[
        tablestore.FieldSchema(
            "type",  # non-vector fields
            tablestore.FieldType.KEYWORD,
            index=True,
            enable_sort_and_agg=True,
        ),
        tablestore.FieldSchema(
            "time",  # non-vector fields
            tablestore.FieldType.LONG,
            index=True,
            enable_sort_and_agg=True,
        ),
    ],
)

TiDB

from llama_index.vector_stores.tidbvector import TiDBVectorStore

tidbvec = TiDBVectorStore(
    # connection url format
    # - mysql+pymysql://[email protected]:4000/test
    connection_string="PLACEHOLDER URL",
    table_name="llama_index_vectorstore",
    distance_strategy="cosine",
    vector_dimension=1536,
)

Timescale

from llama_index.vector_stores.timescalevector import TimescaleVectorStore

vector_store = TimescaleVectorStore.from_params(
    service_url="YOUR TIMESCALE SERVICE URL",
    table_name="paul_graham_essay",
)

Upstash

from llama_index.vector_stores.upstash import UpstashVectorStore

vector_store = UpstashVectorStore(url="YOUR_URL", token="YOUR_TOKEN")

Vertex AI Vector Search

from llama_index.vector_stores.vertexaivectorsearch import VertexAIVectorStore

vector_store = VertexAIVectorStore(
    project_id="[your-google-cloud-project-id]",
    region="[your-google-cloud-region]",
    index_id="[your-index-resource-name]",
    endpoint_id="[your-index-endpoint-name]",
)

Web

import weaviate
from llama_index.vector_stores.weaviate import WeaviateVectorStore

# creating a Weaviate client
resource_owner_config = weaviate.AuthClientPassword(
    username="<username>",
    password="<password>",
)
client = weaviate.Client(
    "https://<cluster-id>.semi.network/",
    auth_client_secret=resource_owner_config,
)

# construct vector store
vector_store = WeaviateVectorStore(weaviate_client=client)

Zulip

Zep 存储文本、元数据和嵌入。所有这些都在搜索结果中返回。

from llama_index.vector_stores.zep import ZepVectorStore

vector_store = ZepVectorStore(
    api_url="<api_url>",
    api_key="<api_key>",
    collection_name="<unique_collection_name>",  # Can either be an existing collection or a new one
    embedding_dimensions=1536,  # Optional, required if creating a new collection
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# Query index using both a text query and metadata filters
filters = MetadataFilters(
    filters=[ExactMatchFilter(key="theme", value="Mafia")]
)
retriever = index.as_retriever(filters=filters)
result = retriever.retrieve("What is inception about?")

Zilliz

• Zilliz Cloud (Milvus 的托管版本) 使用 Milvus Index，并带有一些额外的参数。

import pymilvus
from llama_index.vector_stores.milvus import MilvusVectorStore


# construct vector store
vector_store = MilvusVectorStore(
    uri="foo.vectordb.zillizcloud.com",
    token="your_token_here",
    overwrite="True",
)

示例 Notebook 可以在这里找到.

使用数据连接器从向量存储加载数据

LlamaIndex 支持从大量来源加载数据。有关更多详细信息和 API 文档，请参阅数据连接器。

AlloyDB 存储文档和向量。本教程演示了同步接口。所有同步方法都有相应的异步方法。这是一个如何使用 AlloyDB 的示例

pip install llama-index
pip install llama-index-alloydb-pg

from llama_index.core import SummaryIndex
from llama_index_alloydb_pg import AlloyDBEngine, AlloyDBReader

engine = AlloyDBEngine.from_instance(
    project_id=PROJECT_ID,
    region=REGION,
    cluster=CLUSTER,
    instance=INSTANCE,
    database=DATABASE,
    user=USER,
    password=PASSWORD,
)
reader = AlloyDBReader.create_sync(
    engine,
    table_name=TABLE_NAME,
)
documents = reader.load_data()

index = SummaryIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("<query_text>")
display(Markdown(f"<b>{response}</b>"))

Google Cloud SQL for PostgreSQL 存储文档和向量。本教程演示了同步接口。所有同步方法都有相应的异步方法。这是一个如何使用 Cloud SQL for PostgreSQL 的示例

pip install llama-index
pip install llama-index-cloud-sql-pg

from llama_index.core import SummaryIndex
from llama_index_cloud_sql_pg import PostgresEngine, PostgresReader

engine = PostgresEngine.from_instance(
    project_id=PROJECT_ID,
    region=REGION,
    instance=INSTANCE,
    database=DATABASE,
    user=USER,
    password=PASSWORD,
)
reader = PostgresReader.create_sync(
    engine,
    table_name=TABLE_NAME,
)
documents = reader.load_data()

index = SummaryIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("<query_text>")
display(Markdown(f"<b>{response}</b>"))

Chroma 存储文档和向量。这是一个如何使用 Chroma 的示例

from llama_index.readers.chroma import ChromaReader
from llama_index.core import SummaryIndex

# The chroma reader loads data from a persisted Chroma collection.
# This requires a collection name and a persist directory.
reader = ChromaReader(
    collection_name="chroma_collection",
    persist_directory="examples/data_connectors/chroma_collection",
)

query_vector = [n1, n2, n3, ...]

documents = reader.load_data(
    collection_name="demo", query_vector=query_vector, limit=5
)
index = SummaryIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("<query_text>")
display(Markdown(f"<b>{response}</b>"))

Qdrant 也存储文档和向量。这是一个如何使用 Qdrant 的示例

from llama_index.readers.qdrant import QdrantReader

reader = QdrantReader(host="localhost")

# the query_vector is an embedding representation of your query_vector
# Example query_vector
# query_vector = [0.3, 0.3, 0.3, 0.3, ...]

query_vector = [n1, n2, n3, ...]

# NOTE: Required args are collection_name, query_vector.
# See the Python client: https;//github.com/qdrant/qdrant_client
# for more details

documents = reader.load_data(
    collection_name="demo", query_vector=query_vector, limit=5
)

注意：由于 Weaviate 可以存储文档和向量对象的混合，用户可以选择显式指定 class_name 和 properties 来查询文档，或者他们可以选择指定原始 GraphQL 查询。有关用法请参见下文。

# option 1: specify class_name and properties

# 1) load data using class_name and properties
documents = reader.load_data(
    class_name="<class_name>",
    properties=["property1", "property2", "..."],
    separate_documents=True,
)

# 2) example GraphQL query
query = """
{
    Get {
        <class_name> {
            <property1>
            <property2>
        }
    }
}
"""

documents = reader.load_data(graphql_query=query, separate_documents=True)

注意：Pinecone 和 Faiss 数据加载器都假定相应的数据源只存储向量；文本内容存储在其他地方。因此，这两种数据加载器都要求用户在 load_data 调用中指定 id_to_text_map。

例如，这是 Pinecone 数据加载器 PineconeReader 的一个示例用法

from llama_index.readers.pinecone import PineconeReader

reader = PineconeReader(api_key=api_key, environment="us-west1-gcp")

id_to_text_map = {
    "id1": "text blob 1",
    "id2": "text blob 2",
}

query_vector = [n1, n2, n3, ...]

documents = reader.load_data(
    index_name="quickstart",
    id_to_text_map=id_to_text_map,
    top_k=3,
    vector=query_vector,
    separate_documents=True,
)

示例 Notebook 可以在这里找到.

向量存储示例

• 阿里云 OpenSearch
• Amazon Neptune - Neptune Analytics
• Astra DB
• 异步索引创建
• Azure AI Search
• Azure Cosmos DB
• Caasandra
• Chromadb
• Couchdb
• Dash
• Discord
• DocArray HNSW
• 内存中的 DocArray
• Espilla
• Google AlloyDB for PostgreSQL
• Google Cloud SQL for PostgreSQL
• LanceDB
• Lantern
• Microsoft onedrive
• Minio
• Milvus 异步 API
• Milvus 全文搜索
• Milvus 混合搜索
• MyScale
• Elasticsearch
• FAISS
• MongoDB Atlas
• Neo4j
• OpenSearch
• Pinecone
• Pinecone 混合搜索
• PGvectoRS
• Postgres
• Redis
• Quip
• Qdrant 混合搜索
• Rockset
• Simple
• Supabase
• Tablestore
• Tair
• Tencent
• Timescale
• Upstash
• Web
• Weaviate 混合搜索
• WordLift
• Zulip