自定义存储#

默认情况下，LlamaIndex 会隐藏复杂性，让您可以在不到 5 行代码内查询您的数据

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the documents.")

在底层，LlamaIndex 还支持一个可替换的存储层，允许您自定义存储摄取的文档（即 Node 对象）、嵌入向量和索引元数据的位置。

低级API#

为此，不使用高级API，

index = VectorStoreIndex.from_documents(documents)

我们使用提供更精细控制的低级API

from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.storage.index_store import SimpleIndexStore
from llama_index.core.vector_stores import SimpleVectorStore
from llama_index.core.node_parser import SentenceSplitter

# create parser and parse document into nodes
parser = SentenceSplitter()
nodes = parser.get_nodes_from_documents(documents)

# create storage context using default stores
storage_context = StorageContext.from_defaults(
    docstore=SimpleDocumentStore(),
    vector_store=SimpleVectorStore(),
    index_store=SimpleIndexStore(),
)

# create (or load) docstore and add nodes
storage_context.docstore.add_documents(nodes)

# build index
index = VectorStoreIndex(nodes, storage_context=storage_context)

# save index
index.storage_context.persist(persist_dir="<persist_dir>")

# can also set index_id to save multiple indexes to the same folder
index.set_index_id("<index_id>")
index.storage_context.persist(persist_dir="<persist_dir>")

# to load index later, make sure you setup the storage context
# this will loaded the persisted stores from persist_dir
storage_context = StorageContext.from_defaults(persist_dir="<persist_dir>")

# then load the index object
from llama_index.core import load_index_from_storage

loaded_index = load_index_from_storage(storage_context)

# if loading an index from a persist_dir containing multiple indexes
loaded_index = load_index_from_storage(storage_context, index_id="<index_id>")

# if loading multiple indexes from a persist dir
loaded_indicies = load_index_from_storage(
    storage_context, index_ids=["<index_id>", ...]
)

您可以通过一行代码更改来实例化不同的文档存储、索引存储和向量存储，从而自定义底层存储。详情请参阅文档存储、向量存储、索引存储指南。

向量存储集成和存储#

我们的大多数向量存储集成将整个索引（向量+文本）存储在向量存储本身中。这带来了主要的好处，即无需像上面所示那样显式持久化索引，因为向量存储已经托管并持久化了我们索引中的数据。

支持这种做法的向量存储包括

AzureAISearchVectorStore
ChatGPTRetrievalPluginClient
CassandraVectorStore
ChromaVectorStore
EpsillaVectorStore
DocArrayHnswVectorStore
DocArrayInMemoryVectorStore
JaguarVectorStore
LanceDBVectorStore
MetalVectorStore
MilvusVectorStore
MyScaleVectorStore
OpensearchVectorStore
PineconeVectorStore
QdrantVectorStore
TablestoreVectorStore
RedisVectorStore
UpstashVectorStore
WeaviateVectorStore

下面是一个使用 Pinecone 的小示例

import pinecone
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.pinecone import PineconeVectorStore

# Creating a Pinecone index
api_key = "api_key"
pinecone.init(api_key=api_key, environment="us-west1-gcp")
pinecone.create_index(
    "quickstart", dimension=1536, metric="euclidean", pod_type="p1"
)
index = pinecone.Index("quickstart")

# construct vector store
vector_store = PineconeVectorStore(pinecone_index=index)

# create storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# load documents
documents = SimpleDirectoryReader("./data").load_data()

# create index, which will insert documents/vectors to pinecone
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

如果您有一个已加载数据的现有向量存储，您可以连接到它并直接按如下方式创建 VectorStoreIndex

index = pinecone.Index("quickstart")
vector_store = PineconeVectorStore(pinecone_index=index)
loaded_index = VectorStoreIndex.from_vector_store(vector_store=vector_store)