Weaviate向量存储 - 混合搜索¶

如果您在colab中打开此Notebook，您可能需要安装LlamaIndex 🦙。

In [ ]

%pip install llama-index-vector-stores-weaviate

%pip install llama-index-vector-stores-weaviate

In [ ]

!pip install llama-index

!pip install llama-index

In [ ]

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

import logging import sys logging.basicConfig(stream=sys.stdout, level=logging.INFO) logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

创建Weaviate客户端¶

In [ ]

import os
import openai

os.environ["OPENAI_API_KEY"] = ""
openai.api_key = os.environ["OPENAI_API_KEY"]

import os import openai os.environ["OPENAI_API_KEY"] = "" openai.api_key = os.environ["OPENAI_API_KEY"]

In [ ]

import weaviate

import weaviate

In [ ]

# Connect to cloud instance
cluster_url = ""
api_key = ""

client = weaviate.connect_to_wcs(
    cluster_url=cluster_url,
    auth_credentials=weaviate.auth.AuthApiKey(api_key),
)

# Connect to local instance
# client = weaviate.connect_to_local()

# Connect to cloud instance cluster_url = "" api_key = "" client = weaviate.connect_to_wcs( cluster_url=cluster_url, auth_credentials=weaviate.auth.AuthApiKey(api_key), ) # Connect to local instance # client = weaviate.connect_to_local()

In [ ]

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.weaviate import WeaviateVectorStore
from llama_index.core.response.notebook_utils import display_response

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader from llama_index.vector_stores.weaviate import WeaviateVectorStore from llama_index.core.response.notebook_utils import display_response

下载数据¶

In [ ]

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

加载文档¶

In [ ]

# load documents
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# load documents documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

使用WeaviateVectorStore构建VectorStoreIndex¶

In [ ]

from llama_index.core import StorageContext


vector_store = WeaviateVectorStore(weaviate_client=client)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# NOTE: you may also choose to define a index_name manually.
# index_name = "test_prefix"
# vector_store = WeaviateVectorStore(weaviate_client=client, index_name=index_name)

from llama_index.core import StorageContext vector_store = WeaviateVectorStore(weaviate_client=client) storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents( documents, storage_context=storage_context ) # NOTE: you may also choose to define a index_name manually. # index_name = "test_prefix" # vector_store = WeaviateVectorStore(weaviate_client=client, index_name=index_name)

使用默认向量搜索查询索引¶

In [ ]

# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine(similarity_top_k=2)
response = query_engine.query("What did the author do growing up?")

# set Logging to DEBUG for more detailed outputs query_engine = index.as_query_engine(similarity_top_k=2) response = query_engine.query("What did the author do growing up?")

In [ ]

display_response(response)

display_response(response)

使用混合搜索查询索引¶

使用BM25和向量进行混合搜索。
alpha 参数决定权重（alpha = 0 -> BM25，alpha = 1 -> 向量搜索）。

默认情况下，使用alpha=0.75（非常接近向量搜索）¶

In [ ]

# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine(
    vector_store_query_mode="hybrid", similarity_top_k=2
)
response = query_engine.query(
    "What did the author do growing up?",
)

# set Logging to DEBUG for more detailed outputs query_engine = index.as_query_engine( vector_store_query_mode="hybrid", similarity_top_k=2 ) response = query_engine.query( "What did the author do growing up?", )

In [ ]

display_response(response)

display_response(response)

设置alpha=0.以倾向于BM25¶

In [ ]

# set Logging to DEBUG for more detailed outputs
query_engine = index.as_query_engine(
    vector_store_query_mode="hybrid", similarity_top_k=2, alpha=0.0
)
response = query_engine.query(
    "What did the author do growing up?",
)

# set Logging to DEBUG for more detailed outputs query_engine = index.as_query_engine( vector_store_query_mode="hybrid", similarity_top_k=2, alpha=0.0 ) response = query_engine.query( "What did the author do growing up?", )

In [ ]

display_response(response)

display_response(response)