元数据替换 + 节点句子窗口¶

在本 Notebook 中，我们使用 SentenceWindowNodeParser 将文档解析为每个节点一个句子。每个节点还包含一个“窗口”，其中包含节点句子两侧的句子。

然后，在检索之后，在将检索到的句子传递给 LLM 之前，使用 MetadataReplacementNodePostProcessor 将单个句子替换为包含周围句子的窗口。

这对于大型文档/索引最有用，因为它有助于检索更细粒度的细节。

默认情况下，句子窗口是原始句子两侧的 5 个句子。

在这种情况下，不使用分块大小设置，而是遵循窗口设置。

In [ ]

%pip install llama-index-embeddings-openai
%pip install llama-index-embeddings-huggingface
%pip install llama-index-llms-openai

%pip install llama-index-embeddings-openai %pip install llama-index-embeddings-huggingface %pip install llama-index-llms-openai

In [ ]

%load_ext autoreload
%autoreload 2

%load_ext autoreload %autoreload 2

设置¶

如果你在 Colab 上打开此 Notebook，你可能需要安装 LlamaIndex 🦙。

In [ ]

!pip install llama-index

!pip install llama-index

In [ ]

import os
import openai

import os import openai

In [ ]

os.environ["OPENAI_API_KEY"] = "sk-..."

os.environ["OPENAI_API_KEY"] = "sk-..."

In [ ]

from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.core.node_parser import SentenceSplitter

# create the sentence window node parser w/ default settings
node_parser = SentenceWindowNodeParser.from_defaults(
    window_size=3,
    window_metadata_key="window",
    original_text_metadata_key="original_text",
)

# base node parser is a sentence splitter
text_splitter = SentenceSplitter()

llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)
embed_model = HuggingFaceEmbedding(
    model_name="sentence-transformers/all-mpnet-base-v2", max_length=512
)

from llama_index.core import Settings

Settings.llm = llm
Settings.embed_model = embed_model
Settings.text_splitter = text_splitter

from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.embeddings.huggingface import HuggingFaceEmbedding from llama_index.core.node_parser import SentenceWindowNodeParser from llama_index.core.node_parser import SentenceSplitter # create the sentence window node parser w/ default settings node_parser = SentenceWindowNodeParser.from_defaults( window_size=3, window_metadata_key="window", original_text_metadata_key="original_text", ) # base node parser is a sentence splitter text_splitter = SentenceSplitter() llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1) embed_model = HuggingFaceEmbedding( model_name="sentence-transformers/all-mpnet-base-v2", max_length=512 ) from llama_index.core import Settings Settings.llm = llm Settings.embed_model = embed_model Settings.text_splitter = text_splitter

加载数据，构建索引¶

本节加载数据并构建向量索引。

加载数据¶

在这里，我们使用最近 IPCC 气候报告的第三章构建索引。

In [ ]

!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf

!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: www..ch

In [ ]

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader(
    input_files=["./IPCC_AR6_WGII_Chapter03.pdf"]
).load_data()

from llama_index.core import SimpleDirectoryReader documents = SimpleDirectoryReader( input_files=["./IPCC_AR6_WGII_Chapter03.pdf"] ).load_data()

提取节点¶

我们提取将存储在 VectorIndex 中的节点集。这包括使用句子窗口解析器提取的节点，以及使用标准解析器提取的“基本”节点。

In [ ]

nodes = node_parser.get_nodes_from_documents(documents)

nodes = node_parser.get_nodes_from_documents(documents)

In [ ]

base_nodes = text_splitter.get_nodes_from_documents(documents)

base_nodes = text_splitter.get_nodes_from_documents(documents)

构建索引¶

我们构建句子索引和“基本”索引（使用默认分块大小）。

In [ ]

from llama_index.core import VectorStoreIndex

sentence_index = VectorStoreIndex(nodes)

from llama_index.core import VectorStoreIndex sentence_index = VectorStoreIndex(nodes)

In [ ]

base_index = VectorStoreIndex(base_nodes)

base_index = VectorStoreIndex(base_nodes)

查询¶

使用 MetadataReplacementPostProcessor¶

在这里，我们现在使用 MetadataReplacementPostProcessor 将每个节点中的句子替换为其周围的上下文。

In [ ]

from llama_index.core.postprocessor import MetadataReplacementPostProcessor

query_engine = sentence_index.as_query_engine(
    similarity_top_k=2,
    # the target key defaults to `window` to match the node_parser's default
    node_postprocessors=[
        MetadataReplacementPostProcessor(target_metadata_key="window")
    ],
)
window_response = query_engine.query(
    "What are the concerns surrounding the AMOC?"
)
print(window_response)

from llama_index.core.postprocessor import MetadataReplacementPostProcessor query_engine = sentence_index.as_query_engine( similarity_top_k=2, # the target key defaults to `window` to match the node_parser's default node_postprocessors=[ MetadataReplacementPostProcessor(target_metadata_key="window") ], ) window_response = query_engine.query( "What are the concerns surrounding the AMOC?" ) print(window_response)

There is low confidence in the quantification of Atlantic Meridional Overturning Circulation (AMOC) changes in the 20th century due to low agreement in quantitative reconstructed and simulated trends. Additionally, direct observational records since the mid-2000s remain too short to determine the relative contributions of internal variability, natural forcing, and anthropogenic forcing to AMOC change. However, it is very likely that AMOC will decline for all SSP scenarios over the 21st century, but it will not involve an abrupt collapse before 2100.

我们还可以检查每个节点检索到的原始句子，以及发送给 LLM 的实际句子窗口。

In [ ]

window = window_response.source_nodes[0].node.metadata["window"]
sentence = window_response.source_nodes[0].node.metadata["original_text"]

print(f"Window: {window}")
print("------------------")
print(f"Original Sentence: {sentence}")

window = window_response.source_nodes[0].node.metadata["window"] sentence = window_response.source_nodes[0].node.metadata["original_text"] print(f"Window: {window}") print("------------------") print(f"Original Sentence: {sentence}")

Window: Nevertheless, projected future annual cumulative upwelling wind 
changes at most locations and seasons remain within ±10–20% of 
present-day values (medium confidence) (WGI AR6 Section  9.2.3.5; 
Fox-Kemper et al., 2021).
 Continuous observation of the Atlantic meridional overturning 
circulation (AMOC) has improved the understanding of its variability 
(Frajka-Williams et  al., 2019), but there is low confidence in the 
quantification of AMOC changes in the 20th century because of low 
agreement in quantitative reconstructed and simulated trends (WGI 
AR6 Sections 2.3.3, 9.2.3.1; Fox-Kemper et al., 2021; Gulev et al., 2021). 
 Direct observational records since the mid-2000s remain too short to 
determine the relative contributions of internal variability, natural 
forcing and anthropogenic forcing to AMOC change (high confidence) 
(WGI AR6 Sections 2.3.3, 9.2.3.1; Fox-Kemper et al., 2021; Gulev et al., 
2021).  Over the 21st century, AMOC will very likely decline for all SSP 
scenarios but will not involve an abrupt collapse before 2100 (WGI 
AR6 Sections 4.3.2, 9.2.3.1; Fox-Kemper et al., 2021; Lee et al., 2021).
 3.2.2.4 Sea Ice Changes
Sea ice is a key driver of polar marine life, hosting unique ecosystems 
and affecting diverse marine organisms and food webs through its 
impact on light penetration and supplies of nutrients and organic 
matter (Arrigo, 2014).  Since the late 1970s, Arctic sea ice area has 
decreased for all months, with an estimated decrease of 2 million km2 
(or 25%) for summer sea ice (averaged for August, September and 
October) in 2010–2019 as compared with 1979–1988 (WGI AR6 
Section 9.3.1.1; Fox-Kemper et al., 2021). 
------------------
Original Sentence: Over the 21st century, AMOC will very likely decline for all SSP 
scenarios but will not involve an abrupt collapse before 2100 (WGI 
AR6 Sections 4.3.2, 9.2.3.1; Fox-Kemper et al., 2021; Lee et al., 2021).

与普通 VectorStoreIndex 对比¶

In [ ]

query_engine = base_index.as_query_engine(similarity_top_k=2)
vector_response = query_engine.query(
    "What are the concerns surrounding the AMOC?"
)
print(vector_response)

query_engine = base_index.as_query_engine(similarity_top_k=2) vector_response = query_engine.query( "What are the concerns surrounding the AMOC?" ) print(vector_response)

The concerns surrounding the AMOC are not provided in the given context information.

好吧，那没奏效。我们提高 top k！这将比句子窗口索引更慢，并使用更多 token。

In [ ]

query_engine = base_index.as_query_engine(similarity_top_k=5)
vector_response = query_engine.query(
    "What are the concerns surrounding the AMOC?"
)
print(vector_response)

query_engine = base_index.as_query_engine(similarity_top_k=5) vector_response = query_engine.query( "What are the concerns surrounding the AMOC?" ) print(vector_response)

There are concerns surrounding the AMOC (Atlantic Meridional Overturning Circulation). The context information mentions that the AMOC will decline over the 21st century, with high confidence but low confidence for quantitative projections.

分析¶

所以 SentenceWindowNodeParser + MetadataReplacementNodePostProcessor 组合在这里是明显的赢家。但为什么呢？

句子级别的嵌入似乎捕获了更细粒度的细节，比如“AMOC”这个词。

我们还可以比较每个索引检索到的分块！

In [ ]

for source_node in window_response.source_nodes:
    print(source_node.node.metadata["original_text"])
    print("--------")

for source_node in window_response.source_nodes: print(source_node.node.metadata["original_text"]) print("--------")

Over the 21st century, AMOC will very likely decline for all SSP 
scenarios but will not involve an abrupt collapse before 2100 (WGI 
AR6 Sections 4.3.2, 9.2.3.1; Fox-Kemper et al., 2021; Lee et al., 2021).

--------
Direct observational records since the mid-2000s remain too short to 
determine the relative contributions of internal variability, natural 
forcing and anthropogenic forcing to AMOC change (high confidence) 
(WGI AR6 Sections 2.3.3, 9.2.3.1; Fox-Kemper et al., 2021; Gulev et al., 
2021). 
--------

在这里，我们可以看到句子窗口索引轻松检索到两个谈论 AMOC 的节点。请记住，嵌入完全基于这里的原始句子，但 LLM 最终也会读取周围的上下文！

现在，让我们尝试分析为什么朴素向量索引失败了。

In [ ]

for node in vector_response.source_nodes:
    print("AMOC mentioned?", "AMOC" in node.node.text)
    print("--------")

for node in vector_response.source_nodes: print("AMOC mentioned?", "AMOC" in node.node.text) print("--------")

AMOC mentioned? False
--------
AMOC mentioned? False
--------
AMOC mentioned? True
--------
AMOC mentioned? False
--------
AMOC mentioned? False
--------

所以索引 [2] 的源节点提到了 AMOC，但是这段文本实际看起来像什么？

In [ ]

print(vector_response.source_nodes[2].node.text)

print(vector_response.source_nodes[2].node.text)

2021; Gulev et al. 
2021)The AMOC will decline over the 21st century 
(high confidence, but low confidence for 
quantitative projections).4.3.2.3, 9.2.3 (Fox-Kemper 
et al. 2021; Lee et al. 
2021)
Sea ice
Arctic sea ice 
changes‘Current Arctic sea ice coverage levels are the 
lowest since at least 1850 for both annual mean 
and late-summer values (high confidence).’2.3.2.1, 9.3.1 (Fox-Kemper 
et al. 2021; Gulev et al. 
2021)‘The Arctic will become practically ice-free in 
September by the end of the 21st century under 
SSP2-4.5, SSP3-7.0 and SSP5-8.5[…](high 
confidence).’4.3.2.1, 9.3.1 (Fox-Kemper 
et al. 2021; Lee et al. 
2021)
Antarctic sea ice 
changesThere is no global significant trend in 
Antarctic sea ice area from 1979 to 2020 (high 
confidence).2.3.2.1, 9.3.2 (Fox-Kemper 
et al. 2021; Gulev et al. 
2021)There is low confidence in model simulations of 
future Antarctic sea ice.9.3.2 (Fox-Kemper et al. 
2021)
Ocean chemistry
Changes in salinityThe ‘large-scale, near-surface salinity contrasts 
have intensified since at least 1950 […] 
(virtually certain).’2.3.3.2, 9.2.2.2 
(Fox-Kemper et al. 2021; 
Gulev et al. 2021)‘Fresh ocean regions will continue to get fresher 
and salty ocean regions will continue to get 
saltier in the 21st century (medium confidence).’9.2.2.2 (Fox-Kemper et al. 
2021)
Ocean acidificationOcean surface pH has declined globally over the 
past four decades (virtually certain).2.3.3.5, 5.3.2.2 (Canadell 
et al. 2021; Gulev et al. 
2021)Ocean surface pH will continue to decrease 
‘through the 21st century, except for the 
lower-emission scenarios SSP1-1.9 and SSP1-2.6 
[…] (high confidence).’4.3.2.5, 4.5.2.2, 5.3.4.1 
(Lee et al. 2021; Canadell 
et al. 2021)
Ocean 
deoxygenationDeoxygenation has occurred in most open 
ocean regions since the mid-20th century (high 
confidence).2.3.3.6, 5.3.3.2 (Canadell 
et al. 2021; Gulev et al. 
2021)Subsurface oxygen content ‘is projected to 
transition to historically unprecedented condition 
with decline over the 21st century (medium 
confidence).’5.3.3.2 (Canadell et al. 
2021)
Changes in nutrient 
concentrationsNot assessed in WGI Not assessed in WGI

所以讨论了 AMOC，但遗憾的是它在中间分块中。对于 LLMs，通常观察到检索到的上下文中间的文本常常被忽略或作用较小。最近的一篇论文 “Lost in the Middle”在此讨论了这一点。

[可选] 评估¶

我们更严格地评估句子窗口检索器与基本检索器的效果。

我们定义/加载一个评估基准数据集，然后对其运行不同的评估。

警告: 这可能会很昂贵，特别是使用 GPT-4。请谨慎使用，并调整样本大小以适应你的预算。

In [ ]

from llama_index.core.evaluation import DatasetGenerator, QueryResponseDataset

from llama_index.llms.openai import OpenAI
import nest_asyncio
import random

nest_asyncio.apply()

from llama_index.core.evaluation import DatasetGenerator, QueryResponseDataset from llama_index.llms.openai import OpenAI import nest_asyncio import random nest_asyncio.apply()

In [ ]

len(base_nodes)

len(base_nodes)

Out[ ]

428

In [ ]

num_nodes_eval = 30
# there are 428 nodes total. Take the first 200 to generate questions (the back half of the doc is all references)
sample_eval_nodes = random.sample(base_nodes[:200], num_nodes_eval)
# NOTE: run this if the dataset isn't already saved
# generate questions from the largest chunks (1024)
dataset_generator = DatasetGenerator(
    sample_eval_nodes,
    llm=OpenAI(model="gpt-4"),
    show_progress=True,
    num_questions_per_chunk=2,
)

num_nodes_eval = 30 # there are 428 nodes total. Take the first 200 to generate questions (the back half of the doc is all references) sample_eval_nodes = random.sample(base_nodes[:200], num_nodes_eval) # NOTE: run this if the dataset isn't already saved # generate questions from the largest chunks (1024) dataset_generator = DatasetGenerator( sample_eval_nodes, llm=OpenAI(model="gpt-4"), show_progress=True, num_questions_per_chunk=2, )

In [ ]

eval_dataset = await dataset_generator.agenerate_dataset_from_nodes()

eval_dataset = await dataset_generator.agenerate_dataset_from_nodes()

In [ ]

eval_dataset.save_json("data/ipcc_eval_qr_dataset.json")

eval_dataset.save_json("data/ipcc_eval_qr_dataset.json")

In [ ]

# optional
eval_dataset = QueryResponseDataset.from_json("data/ipcc_eval_qr_dataset.json")

# optional eval_dataset = QueryResponseDataset.from_json("data/ipcc_eval_qr_dataset.json")

比较结果¶

In [ ]

import asyncio
import nest_asyncio

nest_asyncio.apply()

import asyncio import nest_asyncio nest_asyncio.apply()

In [ ]

from llama_index.core.evaluation import (
    CorrectnessEvaluator,
    SemanticSimilarityEvaluator,
    RelevancyEvaluator,
    FaithfulnessEvaluator,
    PairwiseComparisonEvaluator,
)


from collections import defaultdict
import pandas as pd

# NOTE: can uncomment other evaluators
evaluator_c = CorrectnessEvaluator(llm=OpenAI(model="gpt-4"))
evaluator_s = SemanticSimilarityEvaluator()
evaluator_r = RelevancyEvaluator(llm=OpenAI(model="gpt-4"))
evaluator_f = FaithfulnessEvaluator(llm=OpenAI(model="gpt-4"))
# pairwise_evaluator = PairwiseComparisonEvaluator(llm=OpenAI(model="gpt-4"))

from llama_index.core.evaluation import ( CorrectnessEvaluator, SemanticSimilarityEvaluator, RelevancyEvaluator, FaithfulnessEvaluator, PairwiseComparisonEvaluator, ) from collections import defaultdict import pandas as pd # NOTE: can uncomment other evaluators evaluator_c = CorrectnessEvaluator(llm=OpenAI(model="gpt-4")) evaluator_s = SemanticSimilarityEvaluator() evaluator_r = RelevancyEvaluator(llm=OpenAI(model="gpt-4")) evaluator_f = FaithfulnessEvaluator(llm=OpenAI(model="gpt-4")) # pairwise_evaluator = PairwiseComparisonEvaluator(llm=OpenAI(model="gpt-4"))

In [ ]

from llama_index.core.evaluation.eval_utils import (
    get_responses,
    get_results_df,
)
from llama_index.core.evaluation import BatchEvalRunner

max_samples = 30

eval_qs = eval_dataset.questions
ref_response_strs = [r for (_, r) in eval_dataset.qr_pairs]

# resetup base query engine and sentence window query engine
# base query engine
base_query_engine = base_index.as_query_engine(similarity_top_k=2)
# sentence window query engine
query_engine = sentence_index.as_query_engine(
    similarity_top_k=2,
    # the target key defaults to `window` to match the node_parser's default
    node_postprocessors=[
        MetadataReplacementPostProcessor(target_metadata_key="window")
    ],
)

from llama_index.core.evaluation.eval_utils import ( get_responses, get_results_df, ) from llama_index.core.evaluation import BatchEvalRunner max_samples = 30 eval_qs = eval_dataset.questions ref_response_strs = [r for (_, r) in eval_dataset.qr_pairs] # resetup base query engine and sentence window query engine # base query engine base_query_engine = base_index.as_query_engine(similarity_top_k=2) # sentence window query engine query_engine = sentence_index.as_query_engine( similarity_top_k=2, # the target key defaults to `window` to match the node_parser's default node_postprocessors=[ MetadataReplacementPostProcessor(target_metadata_key="window") ], )

In [ ]

import numpy as np

base_pred_responses = get_responses(
    eval_qs[:max_samples], base_query_engine, show_progress=True
)
pred_responses = get_responses(
    eval_qs[:max_samples], query_engine, show_progress=True
)

pred_response_strs = [str(p) for p in pred_responses]
base_pred_response_strs = [str(p) for p in base_pred_responses]

import numpy as np base_pred_responses = get_responses( eval_qs[:max_samples], base_query_engine, show_progress=True ) pred_responses = get_responses( eval_qs[:max_samples], query_engine, show_progress=True ) pred_response_strs = [str(p) for p in pred_responses] base_pred_response_strs = [str(p) for p in base_pred_responses]

In [ ]

evaluator_dict = {
    "correctness": evaluator_c,
    "faithfulness": evaluator_f,
    "relevancy": evaluator_r,
    "semantic_similarity": evaluator_s,
}
batch_runner = BatchEvalRunner(evaluator_dict, workers=2, show_progress=True)

evaluator_dict = { "correctness": evaluator_c, "faithfulness": evaluator_f, "relevancy": evaluator_r, "semantic_similarity": evaluator_s, } batch_runner = BatchEvalRunner(evaluator_dict, workers=2, show_progress=True)

对忠实度/语义相似度运行评估。

In [ ]

eval_results = await batch_runner.aevaluate_responses(
    queries=eval_qs[:max_samples],
    responses=pred_responses[:max_samples],
    reference=ref_response_strs[:max_samples],
)

eval_results = await batch_runner.aevaluate_responses( queries=eval_qs[:max_samples], responses=pred_responses[:max_samples], reference=ref_response_strs[:max_samples], )

In [ ]

base_eval_results = await batch_runner.aevaluate_responses(
    queries=eval_qs[:max_samples],
    responses=base_pred_responses[:max_samples],
    reference=ref_response_strs[:max_samples],
)

base_eval_results = await batch_runner.aevaluate_responses( queries=eval_qs[:max_samples], responses=base_pred_responses[:max_samples], reference=ref_response_strs[:max_samples], )

In [ ]

results_df = get_results_df(
    [eval_results, base_eval_results],
    ["Sentence Window Retriever", "Base Retriever"],
    ["correctness", "relevancy", "faithfulness", "semantic_similarity"],
)
display(results_df)

results_df = get_results_df( [eval_results, base_eval_results], ["Sentence Window Retriever", "Base Retriever"], ["correctness", "relevancy", "faithfulness", "semantic_similarity"], ) display(results_df)

	名称	正确性	相关性	忠实性	语义相似度
0	句子窗口检索器	4.366667	0.933333	0.933333	0.959583
1	基本检索器	4.216667	0.900000	0.933333	0.958664