使用 LlamaIndex 构建多租户 RAG¶

在本 Notebook 中，您将了解如何使用 LlamaIndex 构建多租户 RAG 系统。

1. 设置
2. 下载数据
3. 加载数据
4. 创建索引
5. 创建摄取流水线
6. 更新元数据并插入文档
7. 为每个用户定义查询引擎
8. 查询

设置¶

您应该确保已安装 llama-index 和 pypdf。

In [ ]

!pip install llama-index pypdf

!pip install llama-index pypdf

设置 OpenAI 密钥¶

In [ ]

import os

os.environ["OPENAI_API_KEY"] = "YOUR OPENAI API KEY"

import os os.environ["OPENAI_API_KEY"] = "YOUR OPENAI API KEY"

In [ ]

from llama_index.core import VectorStoreIndex
from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
from llama_index.core import SimpleDirectoryReader
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter

from IPython.display import HTML

from llama_index.core import VectorStoreIndex from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter from llama_index.core import SimpleDirectoryReader from llama_index.core.ingestion import IngestionPipeline from llama_index.core.node_parser import SentenceSplitter from IPython.display import HTML

下载数据¶

我们将使用论文《An LLM Compiler for Parallel Function Calling》和《Dense X Retrieval: What Retrieval Granularity Should We Use?》进行演示。

In [ ]

!wget --user-agent "Mozilla" "https://arxiv.org/pdf/2312.04511.pdf" -O "llm_compiler.pdf"
!wget --user-agent "Mozilla" "https://arxiv.org/pdf/2312.06648.pdf" -O "dense_x_retrieval.pdf"

!wget --user-agent "Mozilla" "https://arxiv.org/pdf/2312.04511.pdf" -O "llm_compiler.pdf" !wget --user-agent "Mozilla" "https://arxiv.org/pdf/2312.06648.pdf" -O "dense_x_retrieval.pdf"

--2024-01-15 14:29:26--  https://arxiv.org/pdf/2312.04511.pdf
Resolving arxiv.org (arxiv.org)... 151.101.131.42, 151.101.67.42, 151.101.3.42, ...
Connecting to arxiv.org (arxiv.org)|151.101.131.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 755837 (738K) [application/pdf]
Saving to: ‘llm_compiler.pdf’


llm_compiler.pdf      0%[                    ]       0  --.-KB/s               
llm_compiler.pdf    100%[===================>] 738.12K  --.-KB/s    in 0.004s  

2024-01-15 14:29:26 (163 MB/s) - ‘llm_compiler.pdf’ saved [755837/755837]

--2024-01-15 14:29:26--  https://arxiv.org/pdf/2312.06648.pdf
Resolving arxiv.org (arxiv.org)... 151.101.131.42, 151.101.67.42, 151.101.3.42, ...
Connecting to arxiv.org (arxiv.org)|151.101.131.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1103758 (1.1M) [application/pdf]
Saving to: ‘dense_x_retrieval.pdf’

dense_x_retrieval.p 100%[===================>]   1.05M  --.-KB/s    in 0.005s  

2024-01-15 14:29:26 (208 MB/s) - ‘dense_x_retrieval.pdf’ saved [1103758/1103758]

加载数据¶

In [ ]

reader = SimpleDirectoryReader(input_files=["dense_x_retrieval.pdf"])
documents_jerry = reader.load_data()

reader = SimpleDirectoryReader(input_files=["llm_compiler.pdf"])
documents_ravi = reader.load_data()

reader = SimpleDirectoryReader(input_files=["dense_x_retrieval.pdf"]) documents_jerry = reader.load_data() reader = SimpleDirectoryReader(input_files=["llm_compiler.pdf"]) documents_ravi = reader.load_data()

创建一个空索引¶

In [ ]

index = VectorStoreIndex.from_documents(documents=[])

index = VectorStoreIndex.from_documents(documents=[])

创建摄取管道¶

In [ ]

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=20),
    ]
)

pipeline = IngestionPipeline( transformations=[ SentenceSplitter(chunk_size=512, chunk_overlap=20), ] )

更新元数据并插入文档¶

In [ ]

for document in documents_jerry:
    document.metadata["user"] = "Jerry"

nodes = pipeline.run(documents=documents_jerry)
# Insert nodes into the index
index.insert_nodes(nodes)

for document in documents_jerry: document.metadata["user"] = "Jerry" nodes = pipeline.run(documents=documents_jerry) # Insert nodes into the index index.insert_nodes(nodes)

In [ ]

for document in documents_ravi:
    document.metadata["user"] = "Ravi"

nodes = pipeline.run(documents=documents_ravi)
# Insert nodes into the index
index.insert_nodes(nodes)

for document in documents_ravi: document.metadata["user"] = "Ravi" nodes = pipeline.run(documents=documents_ravi) # Insert nodes into the index index.insert_nodes(nodes)

定义查询引擎¶

为两个用户定义带有必要过滤器的查询引擎。

In [ ]

# For Jerry
jerry_query_engine = index.as_query_engine(
    filters=MetadataFilters(
        filters=[
            ExactMatchFilter(
                key="user",
                value="Jerry",
            )
        ]
    ),
    similarity_top_k=3,
)

# For Ravi
ravi_query_engine = index.as_query_engine(
    filters=MetadataFilters(
        filters=[
            ExactMatchFilter(
                key="user",
                value="Ravi",
            )
        ]
    ),
    similarity_top_k=3,
)

# For Jerry jerry_query_engine = index.as_query_engine( filters=MetadataFilters( filters=[ ExactMatchFilter( key="user", value="Jerry", ) ] ), similarity_top_k=3, ) # For Ravi ravi_query_engine = index.as_query_engine( filters=MetadataFilters( filters=[ ExactMatchFilter( key="user", value="Ravi", ) ] ), similarity_top_k=3, )

查询¶

In [ ]

# Jerry has Dense X Rerieval paper and should be able to answer following question.
response = jerry_query_engine.query(
    "what are propositions mentioned in the paper?"
)
# Print response
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))

# Jerry 拥有 Dense X Retrieval 论文，应该能够回答以下问题。

{response.response}

'))

该论文提到了命题作为另一种检索单元选择。命题被定义为文本中意义的原子表达式，对应于文本中不同的意义片段。它们是最小的，不能进一步分解成独立的命题。每个命题都是上下文相关的且自包含的，包含文本中解释其意义所需的所有上下文。论文通过比萨斜塔的例子演示了命题的概念，其中一段文字被分解为三个命题，每个命题对应于关于该塔的一个独立事实。

In [ ]

# Ravi has LLMCompiler paper
response = ravi_query_engine.query("what are steps involved in LLMCompiler?")

# Print response
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))

# Ravi 拥有 LLMCompiler 论文

{response.response}

'))

LLMCompiler 由三个关键组件组成：LLM规划器、任务获取单元和执行器。LLM规划器通过根据用户输入定义不同的函数调用及其依赖关系来确定执行流程。任务获取单元在用前置任务的实际输出替换变量后，调度可以并行执行的函数调用。最后，执行器使用关联的工具执行已调度的函数调用任务。这些组件协同工作，以优化 LLM 的并行函数调用性能。

In [ ]

# This should not be answered as Jerry does not have information about LLMCompiler
response = jerry_query_engine.query("what are steps involved in LLMCompiler?")

# Print response
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))

# Jerry 没有 LLMCompiler 的信息，因此不应回答此问题。

{response.response}

'))

给定上下文信息中未提及 LLMCompiler 中涉及的步骤。