评估查询引擎工具¶

在本节中，我们将向您展示如何将 `EvalQueryEngineTool` 与代理一起使用。您可能希望使用 `EvalQueryEngineTool` 的一些原因：

1. 为工具使用特定类型的评估，而不仅仅是代理的推理
2. 使用与代理 LLM 不同的 LLM 来评估工具响应

`EvalQueryEngineTool` 构建在 `QueryEngineTool` 之上。除了包装现有的 查询引擎 之外，它还必须提供现有的 评估器 来评估该查询引擎的响应。

安装依赖¶

In [ ]

%pip install llama-index-embeddings-huggingface
%pip install llama-index-llms-openai
%pip install llama-index-agents-openai

%pip install llama-index-embeddings-huggingface %pip install llama-index-llms-openai %pip install llama-index-agents-openai

In [ ]

import os

os.environ["OPENAI_API_KEY"] = "sk-..."

import os os.environ["OPENAI_API_KEY"] = "sk-..."

初始化并设置 LLM 和本地嵌入模型¶

In [ ]

from llama_index.core.settings import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.openai import OpenAI

Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)
Settings.llm = OpenAI()

from llama_index.core.settings import Settings from llama_index.embeddings.huggingface import HuggingFaceEmbedding from llama_index.llms.openai import OpenAI Settings.embed_model = HuggingFaceEmbedding( model_name="BAAI/bge-small-en-v1.5" ) Settings.llm = OpenAI()

下载并索引数据¶

这是我们为了演示目的而进行的。在生产环境中，数据存储和索引应该已经存在，而不是临时创建。

创建存储上下文¶

In [ ]

from llama_index.core import (
    StorageContext,
    load_index_from_storage,
)

try:
    storage_context = StorageContext.from_defaults(
        persist_dir="./storage/lyft",
    )
    lyft_index = load_index_from_storage(storage_context)

    storage_context = StorageContext.from_defaults(
        persist_dir="./storage/uber"
    )
    uber_index = load_index_from_storage(storage_context)

    index_loaded = True
except:
    index_loaded = False

from llama_index.core import ( StorageContext, load_index_from_storage, ) try: storage_context = StorageContext.from_defaults( persist_dir="./storage/lyft", ) lyft_index = load_index_from_storage(storage_context) storage_context = StorageContext.from_defaults( persist_dir="./storage/uber" ) uber_index = load_index_from_storage(storage_context) index_loaded = True except: index_loaded = False

下载数据

In [ ]

!mkdir -p 'data/10k/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf'

!mkdir -p 'data/10k/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf'

加载数据¶

In [ ]

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

if not index_loaded:
    # load data
    lyft_docs = SimpleDirectoryReader(
        input_files=["./data/10k/lyft_2021.pdf"]
    ).load_data()
    uber_docs = SimpleDirectoryReader(
        input_files=["./data/10k/uber_2021.pdf"]
    ).load_data()

    # build index
    lyft_index = VectorStoreIndex.from_documents(lyft_docs)
    uber_index = VectorStoreIndex.from_documents(uber_docs)

    # persist index
    lyft_index.storage_context.persist(persist_dir="./storage/lyft")
    uber_index.storage_context.persist(persist_dir="./storage/uber")

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex if not index_loaded: # load data lyft_docs = SimpleDirectoryReader( input_files=["./data/10k/lyft_2021.pdf"] ).load_data() uber_docs = SimpleDirectoryReader( input_files=["./data/10k/uber_2021.pdf"] ).load_data() # build index lyft_index = VectorStoreIndex.from_documents(lyft_docs) uber_index = VectorStoreIndex.from_documents(uber_docs) # persist index lyft_index.storage_context.persist(persist_dir="./storage/lyft") uber_index.storage_context.persist(persist_dir="./storage/uber")

创建查询引擎¶

In [ ]

lyft_engine = lyft_index.as_query_engine(similarity_top_k=5)
uber_engine = uber_index.as_query_engine(similarity_top_k=5)

lyft_engine = lyft_index.as_query_engine(similarity_top_k=5) uber_engine = uber_index.as_query_engine(similarity_top_k=5)

创建评估器¶

In [ ]

from llama_index.core.evaluation import RelevancyEvaluator

evaluator = RelevancyEvaluator()

from llama_index.core.evaluation import RelevancyEvaluator evaluator = RelevancyEvaluator()

创建查询引擎工具¶

In [ ]

from llama_index.core.tools import ToolMetadata
from llama_index.core.tools.eval_query_engine import EvalQueryEngineTool

query_engine_tools = [
    EvalQueryEngineTool(
        evaluator=evaluator,
        query_engine=lyft_engine,
        metadata=ToolMetadata(
            name="lyft",
            description=(
                "Provides information about Lyft's financials for year 2021. "
                "Use a detailed plain text question as input to the tool."
            ),
        ),
    ),
    EvalQueryEngineTool(
        evaluator=evaluator,
        query_engine=uber_engine,
        metadata=ToolMetadata(
            name="uber",
            description=(
                "Provides information about Uber's financials for year 2021. "
                "Use a detailed plain text question as input to the tool."
            ),
        ),
    ),
]

from llama_index.core.tools import ToolMetadata from llama_index.core.tools.eval_query_engine import EvalQueryEngineTool query_engine_tools = [ EvalQueryEngineTool( evaluator=evaluator, query_engine=lyft_engine, metadata=ToolMetadata( name="lyft", description=( "提供有关 Lyft 2021 年财务信息。使用详细的纯文本问题作为工具的输入。" ), ), ), EvalQueryEngineTool( evaluator=evaluator, query_engine=uber_engine, metadata=ToolMetadata( name="uber", description=( "提供有关 Uber 2021 年财务信息。使用详细的纯文本问题作为工具的输入。" ), ), ), ]

设置 OpenAI 代理¶

In [ ]

from llama_index.agent.openai import OpenAIAgent

agent = OpenAIAgent.from_tools(query_engine_tools, verbose=True)

from llama_index.agent.openai import OpenAIAgent agent = OpenAIAgent.from_tools(query_engine_tools, verbose=True)

查询引擎评估失败¶

为了演示目的，我们将让代理首先选择错误的工具，以便观察 `EvalQueryEngineTool` 在评估失败时的效果。为此，我们在调用代理时将 `tool_choice` 设置为 `lyft`。

这是我们应该预期的结果

1. 代理将首先使用 `lyft` 工具，该工具包含错误的财务信息，这是我们指示它这样做的。
2. `EvalQueryEngineTool` 将使用其评估器评估查询引擎的响应。
3. 查询引擎的输出将因包含 Lyft 的财务信息而非 Uber 的财务信息而评估失败。
4. 工具将形成一个响应，通知代理该工具无法使用，并说明原因。
5. 代理将回退到第二个工具，即 `uber`。
6. 第二个工具的查询引擎输出将通过评估，因为它包含 Uber 的财务信息。
7. 代理将响应一个答案。

In [ ]

response = await agent.achat(
    "What was Uber's revenue growth in 2021?", tool_choice="lyft"
)
print(str(response))

response = await agent.achat( "What was Uber's revenue growth in 2021?", tool_choice="lyft" ) print(str(response))

Added user message to memory: What was Uber's revenue growth in 2021?
=== Calling Function ===
Calling function: lyft with args: {"input":"What was Uber's revenue growth in 2021?"}
Got output: Could not use tool lyft because it failed evaluation.
Reason: NO
========================

=== Calling Function ===
Calling function: uber with args: {"input":"What was Uber's revenue growth in 2021?"}
Got output: Uber's revenue grew by 57% in 2021.
========================

Uber's revenue grew by 57% in 2021.

查询引擎评估通过¶

这里我们询问有关 Lyft 财务信息的问题。这是我们应该预期的结果

1. 代理将首先使用 `lyftk` 工具，仅仅基于其描述，因为我们在这里**未**设置 `tool_choice`。
2. `EvalQueryEngineTool` 将使用其评估器评估查询引擎的响应。
3. 查询引擎的输出将通过评估，因为它包含 Lyft 的财务信息。

In [ ]

response = await agent.achat("What was Lyft's revenue growth in 2021?")
print(str(response))

response = await agent.achat("What was Lyft's revenue growth in 2021?") print(str(response))

Added user message to memory: What was Lyft's revenue growth in 2021?
=== Calling Function ===
Calling function: lyft with args: {"input": "What was Lyft's revenue growth in 2021?"}
Got output: Lyft's revenue growth in 2021 was $3,208,323, which increased compared to the revenue in 2020 and 2019.
========================

=== Calling Function ===
Calling function: uber with args: {"input": "What was Lyft's revenue growth in 2021?"}
Got output: Could not use tool uber because it failed evaluation.
Reason: NO
========================

Lyft's revenue grew by $3,208,323 in 2021, which increased compared to the revenue in 2020 and 2019.