Wandb 回调处理器¶

Weights & Biases Prompts 是一套专为开发 LLM 驱动应用程序而构建的 LLMOps 工具。

WandbCallbackHandler 集成了 W&B Prompts，用于可视化和检查索引构建或索引查询等执行流程。您可以使用此处理器将创建的索引作为 W&B Artifacts 持久化，从而对您的索引进行版本控制。

In [ ]

%pip install llama-index-callbacks-wandb
%pip install llama-index-llms-openai

%pip install llama-index-callbacks-wandb %pip install llama-index-llms-openai

In [ ]

import os
from getpass import getpass

if os.getenv("OPENAI_API_KEY") is None:
    os.environ["OPENAI_API_KEY"] = getpass(
        "Paste your OpenAI key from:"
        " https://platform.openai.com/account/api-keys\n"
    )
assert os.getenv("OPENAI_API_KEY", "").startswith(
    "sk-"
), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")

import os from getpass import getpass if os.getenv("OPENAI_API_KEY") is None: os.environ["OPENAI_API_KEY"] = getpass( "从以下地址粘贴您的 OpenAI 密钥：" " https://platform.openai.com/account/api-keys\n" ) assert os.getenv("OPENAI_API_KEY", "").startswith( "sk-" ), "这看起来不像一个有效的 OpenAI API 密钥" print("OpenAI API 密钥已配置")

OpenAI API key configured

In [ ]

from llama_index.core.callbacks import CallbackManager
from llama_index.core.callbacks import LlamaDebugHandler
from llama_index.callbacks.wandb import WandbCallbackHandler
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    SimpleKeywordTableIndex,
    StorageContext,
)
from llama_index.llms.openai import OpenAI

from llama_index.core.callbacks import CallbackManager from llama_index.core.callbacks import LlamaDebugHandler from llama_index.callbacks.wandb import WandbCallbackHandler from llama_index.core import ( VectorStoreIndex, SimpleDirectoryReader, SimpleKeywordTableIndex, StorageContext, ) from llama_index.llms.openai import OpenAI

设置 LLM¶

In [ ]

from llama_index.core import Settings

Settings.llm = OpenAI(model="gpt-4", temperature=0)

from llama_index.core import Settings Settings.llm = OpenAI(model="gpt-4", temperature=0)

W&B 回调管理器设置¶

选项 1: 设置全局评估处理器

In [ ]

import llama_index.core
from llama_index.core import set_global_handler

set_global_handler("wandb", run_args={"project": "llamaindex"})
wandb_callback = llama_index.core.global_handler

import llama_index.core from llama_index.core import set_global_handler set_global_handler("wandb", run_args={"project": "llamaindex"}) wandb_callback = llama_index.core.global_handler

选项 2: 手动配置回调处理器

同时配置调试处理器以提高 Notebook 可见性。

In [ ]

llama_debug = LlamaDebugHandler(print_trace_on_end=True)

# wandb.init args
run_args = dict(
    project="llamaindex",
)

wandb_callback = WandbCallbackHandler(run_args=run_args)

Settings.callback_manager = CallbackManager([llama_debug, wandb_callback])

llama_debug = LlamaDebugHandler(print_trace_on_end=True) # wandb.init 参数 run_args = dict( project="llamaindex", ) wandb_callback = WandbCallbackHandler(run_args=run_args) Settings.callback_manager = CallbackManager([llama_debug, wandb_callback])

运行上述单元格后，您将获得 W&B 运行页面 URL。在此处，您将找到一个追踪表格，其中包含使用 Weights and Biases' Prompts 功能追踪的所有事件。

1. 索引构建¶

下载数据

In [ ]

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

In [ ]

docs = SimpleDirectoryReader("./data/paul_graham/").load_data()

docs = SimpleDirectoryReader("./data/paul_graham/").load_data()

In [ ]

index = VectorStoreIndex.from_documents(docs)

index = VectorStoreIndex.from_documents(docs)

**********
Trace: index_construction
    |_node_parsing ->  0.295179 seconds
      |_chunking ->  0.293976 seconds
    |_embedding ->  0.494492 seconds
    |_embedding ->  0.346162 seconds
**********

wandb: Logged trace tree to W&B.

1.1 将索引作为 W&B Artifacts 持久化¶

In [ ]

wandb_callback.persist_index(index, index_name="simple_vector_store")

wandb_callback.persist_index(index, index_name="simple_vector_store")

wandb: Adding directory to artifact (/Users/loganmarkewich/llama_index/docs/examples/callbacks/wandb/run-20230801_152955-ds93prxa/files/storage)... Done. 0.0s

1.2 从 W&B Artifacts 下载索引¶

In [ ]

from llama_index.core import load_index_from_storage

storage_context = wandb_callback.load_storage_context(
    artifact_url="ayut/llamaindex/simple_vector_store:v0"
)

# Load the index and initialize a query engine
index = load_index_from_storage(
    storage_context,
)

from llama_index.core import load_index_from_storage storage_context = wandb_callback.load_storage_context( artifact_url="ayut/llamaindex/simple_vector_store:v0" ) # 加载索引并初始化查询引擎 index = load_index_from_storage( storage_context, )

wandb:   3 of 3 files downloaded.  

**********
Trace: index_construction
**********

2. 查询索引¶

In [ ]

query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response, sep="\n")

query_engine = index.as_query_engine() response = query_engine.query("What did the author do growing up?") print(response, sep="\n")

**********
Trace: query
    |_query ->  2.695958 seconds
      |_retrieve ->  0.806379 seconds
        |_embedding ->  0.802871 seconds
      |_synthesize ->  1.8893 seconds
        |_llm ->  1.842434 seconds
**********

wandb: Logged trace tree to W&B.

The text does not provide information on what the author did growing up.

关闭 W&B 回调处理器¶

当我们完成事件追踪后，可以关闭 wandb 运行。

In [ ]

wandb_callback.finish()

wandb_callback.finish()