Timescale 向量存储 (PostgreSQL)¶

本 notebook 演示了如何使用 Postgres 向量存储 TimescaleVector 来存储和查询向量嵌入。

什么是 Timescale Vector？¶

Timescale Vector 是专为 AI 应用打造的 PostgreSQL++。

Timescale Vector 使您能够在 PostgreSQL 中高效存储和查询数百万个向量嵌入。

• 通过受 DiskANN 启发的索引算法，增强了 pgvector，可在数百万个向量上实现更快、更准确的相似性搜索。
• 通过自动基于时间的分割和索引，实现快速基于时间的向量搜索。
• 提供熟悉的 SQL 接口，用于查询向量嵌入和关系数据。

Timescale Vector 伴随你从 POC 扩展到生产环境

• 通过允许您在单个数据库中存储关系元数据、向量嵌入和时间序列数据，简化了操作。
• 得益于坚如磐石的 PostgreSQL 基础，具备流式备份和复制、高可用性和行级安全性等企业级功能。
• 提供企业级安全性和合规性，让您无后顾之忧。

如何使用 Timescale Vector¶

Timescale Vector 可在云端 PostgreSQL 平台 Timescale 上获得。（目前没有自托管版本。）

LlamaIndex 用户可获得 Timescale Vector 90 天免费试用。

• 要开始使用，请[注册](https://console.cloud.timescale.com/signup?utm_campaign=vectorlaunch&utm_source=llamaindex&utm_medium=referral) Timescale，创建一个新数据库并按照本 notebook 操作！
• 有关详细信息和性能基准测试，请参阅 Timescale Vector 解释性博客。
• 有关在 python 中使用 Timescale Vector 的更多详细信息，请参阅安装说明。

0. 设置¶

让我们导入此 notebook 所需的一切。

如果您在 colab 上打开此 Notebook，您可能需要安装 LlamaIndex 🦙。

In [ ]

%pip install llama-index-embeddings-openai
%pip install llama-index-vector-stores-timescalevector

%pip install llama-index-embeddings-openai %pip install llama-index-vector-stores-timescalevector

In [ ]

!pip install llama-index

!pip install llama-index

In [ ]

# import logging
# import sys

# Uncomment to see debug logs
# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

import timescale_vector
from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.vector_stores.timescalevector import TimescaleVectorStore
from llama_index.core.vector_stores import VectorStoreQuery, MetadataFilters
import textwrap
import openai

# import logging # import sys # Uncomment to see debug logs # logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout)) import timescale_vector from llama_index.core import SimpleDirectoryReader, StorageContext from llama_index.core import VectorStoreIndex from llama_index.vector_stores.timescalevector import TimescaleVectorStore from llama_index.core.vector_stores import VectorStoreQuery, MetadataFilters import textwrap import openai

设置 OpenAI API 密钥¶

要为加载到索引中的文档创建嵌入，请配置您的 OpenAI API 密钥

In [ ]

# Get openAI api key by reading local .env file
# The .env file should contain a line starting with `OPENAI_API_KEY=sk-`
import os
from dotenv import load_dotenv, find_dotenv

_ = load_dotenv(find_dotenv())

# OR set it explicitly
# import os
# os.environ["OPENAI_API_KEY"] = "<your key>"
openai.api_key = os.environ["OPENAI_API_KEY"]

# Get openAI api key by reading local .env file # The .env file should contain a line starting with `OPENAI_API_KEY=sk-` import os from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) # OR set it explicitly # import os # os.environ["OPENAI_API_KEY"] = "" openai.api_key = os.environ["OPENAI_API_KEY"]

创建一个 PostgreSQL 数据库并获取 Timescale 服务 URL¶

您需要一个服务 URL 来连接到您的 Timescale 数据库实例。

首先，在 Timescale 中启动一个新的云数据库（使用上面的链接免费注册）。

要连接到您的云 PostgreSQL 数据库，您需要您的服务 URI，可以在创建新数据库后下载的备忘单或 `.env` 文件中找到它。

URI 将类似于：postgres://tsdbadmin:<password>@<id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require

In [ ]

# Get the service url by reading local .env file
# The .env file should contain a line starting with `TIMESCALE_SERVICE_URL=postgresql://`
import os
from dotenv import load_dotenv, find_dotenv

_ = load_dotenv(find_dotenv())

TIMESCALE_SERVICE_URL = os.environ["TIMESCALE_SERVICE_URL"]

# OR set it explicitly
# TIMESCALE_SERVICE_URL = "postgres://tsdbadmin:<password>@<id>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require"

# Get the service url by reading local .env file # The .env file should contain a line starting with `TIMESCALE_SERVICE_URL=postgresql://` import os from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) TIMESCALE_SERVICE_URL = os.environ["TIMESCALE_SERVICE_URL"] # OR set it explicitly # TIMESCALE_SERVICE_URL = "postgres://tsdbadmin@.tsdb.cloud.timescale.com/tsdb?sslmode=require"

1. 使用 Timescale Vector 进行简单相似性搜索¶

下载数据¶

In [ ]

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

加载文档¶

对于本示例，我们将使用 SimpleDirectoryReader 来加载存储在 paul_graham_essay 目录中的文档。

SimpleDirectoryReader 是 LlamaIndex 最常用的数据连接器之一，用于从目录中读取一个或多个文件。

In [ ]

# load sample data from the data directory using a SimpleDirectoryReader
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
print("Document ID:", documents[0].doc_id)

# load sample data from the data directory using a SimpleDirectoryReader documents = SimpleDirectoryReader("./data/paul_graham").load_data() print("Document ID:", documents[0].doc_id)

Document ID: 740ce1a1-4d95-40cc-b7f7-6d2874620a53

使用 TimescaleVectorStore 创建向量存储索引¶

接下来，为了执行相似性搜索，我们首先创建一个 TimescaleVector [向量存储](https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/storage/vector_stores.html) 来存储来自论文内容的向量嵌入。TimescaleVectorStore 接受几个参数，即我们在上面加载的 service_url，以及将用于存储向量的表名的 table_name。

然后我们使用之前的文档在 Timescale 支持的文档上创建一个[向量存储索引](https://gpt-index.readthedocs.io/en/stable/community/integrations/vector_stores.html#vector-store-index)。

In [ ]

# Create a TimescaleVectorStore to store the documents
vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="paul_graham_essay",
)

# Create a new VectorStoreIndex using the TimescaleVectorStore
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# Create a TimescaleVectorStore to store the documents vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="paul_graham_essay", ) # Create a new VectorStoreIndex using the TimescaleVectorStore storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents( documents, storage_context=storage_context )

查询索引¶

现在我们已经在 VectorStore 中索引了文档，我们可以使用默认的 query_engine 来询问索引中的文档相关问题。

注意，您还可以配置查询引擎，以配置返回的最相似结果的 top_k 个数，以及用于过滤结果的元数据过滤器。有关更多详细信息，请参阅[配置标准查询设置部分](https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/index/vector_store_guide.html)。

In [ ]

query_engine = index.as_query_engine()
response = query_engine.query("Did the author work at YC?")

query_engine = index.as_query_engine() response = query_engine.query("Did the author work at YC?")

In [ ]

print(textwrap.fill(str(response), 100))

print(textwrap.fill(str(response), 100))

Yes, the author did work at YC.

In [ ]

response = query_engine.query("What did the author work on before college?")

response = query_engine.query("What did the author work on before college?")

In [ ]

print(textwrap.fill(str(response), 100))

print(textwrap.fill(str(response), 100))

Before college, the author worked on writing and programming. They wrote short stories and also
tried programming on the IBM 1401 computer using an early version of Fortran.

查询现有索引¶

在上面的示例中，我们从加载的文档创建了一个新的 Timescale Vector 向量存储和索引。接下来我们将介绍如何查询现有索引。我们只需要服务 URI 和我们要访问的表名。

In [ ]

vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="paul_graham_essay",
)

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do before YC?")

vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="paul_graham_essay", ) index = VectorStoreIndex.from_vector_store(vector_store=vector_store) query_engine = index.as_query_engine() response = query_engine.query("What did the author do before YC?")

In [ ]

print(textwrap.fill(str(response), 100))

print(textwrap.fill(str(response), 100))

Before YC, the author wrote all of YC's internal software in Arc. They also worked on HN and had
three projects: writing essays, working on YC, and working in Arc. However, they gradually stopped
working on Arc due to time constraints and the increasing dependence on it for infrastructure.

2. 使用 ANN 搜索索引加速查询¶

(注意：这些索引是 ANN 索引，与 LlamaIndex 中的索引概念不同)

您可以通过在嵌入列上创建索引来加速相似性查询。您应该在摄取大部分数据后才执行此操作。

Timescale Vector 支持以下索引

• timescale_vector_index：受 DiskANN 启发的图索引，用于快速相似性搜索（默认）。
• pgvector 的 HNSW 索引：分层可导航小世界图索引，用于快速相似性搜索。
• pgvector 的 IVFFLAT 索引：倒排文件索引，用于快速相似性搜索。

重要提示：在 PostgreSQL 中，每个表在特定列上只能有一个索引。因此，如果您想测试不同索引类型的性能，可以通过以下方式之一进行：(1) 创建多个具有不同索引的表，(2) 在同一个表中创建多个向量列并在每个列上创建不同的索引，或 (3) 通过删除并在同一列上重新创建索引并比较结果。

In [ ]

# Instantiate the TimescaleVectorStore from part 1
vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="paul_graham_essay",
)

# Instantiate the TimescaleVectorStore from part 1 vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="paul_graham_essay", )

默认情况下，使用不带额外参数的 create_index() 函数将创建 timescale_vector (DiskANN) 索引，并使用默认参数。

In [ ]

# Create a timescale vector index (DiskANN)
vector_store.create_index()

# Create a timescale vector index (DiskANN) vector_store.create_index()

您还可以指定索引的参数。有关不同参数及其对性能影响的完整讨论，请参阅 Timescale Vector 文档。

In [ ]

# drop old index
vector_store.drop_index()

# create new timescale vector index (DiskANN) with specified parameters
vector_store.create_index("tsv", max_alpha=1.0, num_neighbors=50)

# drop old index vector_store.drop_index() # create new timescale vector index (DiskANN) with specified parameters vector_store.create_index("tsv", max_alpha=1.0, num_neighbors=50)

Timescale Vector 还支持 HNSW 和 ivfflat 索引

In [ ]

vector_store.drop_index()

# Create an HNSW index
# Note: You don't need to specify m and ef_construction parameters as we set smart defaults.
vector_store.create_index("hnsw", m=16, ef_construction=64)

vector_store.drop_index() # Create an HNSW index # Note: You don't need to specify m and ef_construction parameters as we set smart defaults. vector_store.create_index("hnsw", m=16, ef_construction=64)

In [ ]

# Create an IVFFLAT index
# Note: You don't need to specify num_lists and num_records parameters as we set smart defaults.
vector_store.drop_index()
vector_store.create_index("ivfflat", num_lists=20, num_records=1000)

# Create an IVFFLAT index # Note: You don't need to specify num_lists and num_records parameters as we set smart defaults. vector_store.drop_index() vector_store.create_index("ivfflat", num_lists=20, num_records=1000)

我们通常建议使用 timescale-vector 或 HNSW 索引。

In [ ]

# drop the ivfflat index
vector_store.drop_index()
# Create a timescale vector index (DiskANN)
vector_store.create_index()

# drop the ivfflat index vector_store.drop_index() # Create a timescale vector index (DiskANN) vector_store.create_index()

3. 基于时间过滤的相似性搜索¶

Timescale Vector 的一个关键用例是高效的基于时间的向量搜索。Timescale Vector 通过按时间自动分割向量（及相关元数据）来实现这一点。这使得您能够根据查询向量的相似性和时间来高效地查询向量。

基于时间的向量搜索功能对于以下应用很有帮助，例如

• 存储和检索 LLM 响应历史记录（例如，聊天机器人）
• 查找与查询向量相似的最新嵌入（例如，最新新闻）。
• 将相似性搜索限制在相关时间范围内（例如，就知识库提问基于时间的问题）。

为了说明如何使用 TimescaleVector 基于时间的向量搜索功能，我们将使用 TimescaleDB 的 git 日志历史记录作为示例数据集并提问相关问题。每个 git 提交条目都关联一个时间戳，以及自然语言消息和其他元数据（例如，作者、提交哈希等）。

我们将说明如何使用基于时间的 uuid 创建节点，以及如何使用 TimescaleVector 向量存储运行带时间范围过滤器的相似性搜索。

从 git 日志 CSV 文件中提取内容和元数据¶

首先，让我们将 git 日志 csv 文件加载到我们的 PostgreSQL 数据库中一个名为 timescale_commits 的新集合中。

注意：由于这是一个演示，我们将只处理前 1000 条记录。实际上，您可以加载任意数量的记录。

In [ ]

import pandas as pd
from pathlib import Path

file_path = Path("../data/csv/commit_history.csv")
# Read the CSV file into a DataFrame
df = pd.read_csv(file_path)

# Light data cleaning on CSV
df.dropna(inplace=True)
df = df.astype(str)
df = df[:1000]

import pandas as pd from pathlib import Path file_path = Path("../data/csv/commit_history.csv") # Read the CSV file into a DataFrame df = pd.read_csv(file_path) # Light data cleaning on CSV df.dropna(inplace=True) df = df.astype(str) df = df[:1000]

In [ ]

# Take a look at the data in the csv (optional)
df.head()

# Take a look at the data in the csv (optional) df.head()

我们将定义一个辅助函数，用于根据节点的时间戳为其创建 uuid 和相关的向量嵌入。我们将使用此函数为每个 git 日志条目创建 uuid。

重要提示：如果您正在处理文档/节点，并且希望将当前日期和时间与向量关联起来以进行基于时间的搜索，则可以跳过此步骤。默认情况下，当节点添加到 Timescale Vector 的表中时，将自动生成 uuid。在我们的例子中，因为我们希望 uuid 基于过去的时间戳，所以我们需要手动创建 uuid。

In [ ]

from timescale_vector import client


# Function to take in a date string in the past and return a uuid v1
def create_uuid(date_string: str):
    if date_string is None:
        return None
    time_format = "%a %b %d %H:%M:%S %Y %z"
    datetime_obj = datetime.strptime(date_string, time_format)
    uuid = client.uuid_from_time(datetime_obj)
    return str(uuid)

from timescale_vector import client # Function to take in a date string in the past and return a uuid v1 def create_uuid(date_string: str): if date_string is None: return None time_format = "%a %b %d %H:%M:%S %Y %z" datetime_obj = datetime.strptime(date_string, time_format) uuid = client.uuid_from_time(datetime_obj) return str(uuid)

In [ ]

# Helper functions
from typing import List, Tuple


# Helper function to split name and email given an author string consisting of Name Lastname <email>
def split_name(input_string: str) -> Tuple[str, str]:
    if input_string is None:
        return None, None
    start = input_string.find("<")
    end = input_string.find(">")
    name = input_string[:start].strip()
    return name


from datetime import datetime, timedelta


def create_date(input_string: str) -> datetime:
    if input_string is None:
        return None
    # Define a dictionary to map month abbreviations to their numerical equivalents
    month_dict = {
        "Jan": "01",
        "Feb": "02",
        "Mar": "03",
        "Apr": "04",
        "May": "05",
        "Jun": "06",
        "Jul": "07",
        "Aug": "08",
        "Sep": "09",
        "Oct": "10",
        "Nov": "11",
        "Dec": "12",
    }

    # Split the input string into its components
    components = input_string.split()
    # Extract relevant information
    day = components[2]
    month = month_dict[components[1]]
    year = components[4]
    time = components[3]
    timezone_offset_minutes = int(
        components[5]
    )  # Convert the offset to minutes
    timezone_hours = timezone_offset_minutes // 60  # Calculate the hours
    timezone_minutes = (
        timezone_offset_minutes % 60
    )  # Calculate the remaining minutes
    # Create a formatted string for the timestamptz in PostgreSQL format
    timestamp_tz_str = (
        f"{year}-{month}-{day} {time}+{timezone_hours:02}{timezone_minutes:02}"
    )
    return timestamp_tz_str

# Helper functions from typing import List, Tuple # Helper function to split name and email given an author string consisting of Name Lastnamedef split_name(input_string: str) -> Tuple[str, str]: if input_string is None: return None, None start = input_string.find("<") end = input_string.find(">") name = input_string[:start].strip() return name from datetime import datetime, timedelta def create_date(input_string: str) -> datetime: if input_string is None: return None # Define a dictionary to map month abbreviations to their numerical equivalents month_dict = { "Jan": "01", "Feb": "02", "Mar": "03", "Apr": "04", "May": "05", "Jun": "06", "Jul": "07", "Aug": "08", "Sep": "09", "Oct": "10", "Nov": "11", "Dec": "12", } # Split the input string into its components components = input_string.split() # Extract relevant information day = components[2] month = month_dict[components[1]] year = components[4] time = components[3] timezone_offset_minutes = int( components[5] ) # Convert the offset to minutes timezone_hours = timezone_offset_minutes // 60 # Calculate the hours timezone_minutes = ( timezone_offset_minutes % 60 ) # Calculate the remaining minutes # Create a formatted string for the timestamptz in PostgreSQL format timestamp_tz_str = ( f"{year}-{month}-{day} {time}+{timezone_hours:02}{timezone_minutes:02}" ) return timestamp_tz_str

Next, we'll define a function to create a TextNode for each git log entry. We'll use the helper function create_uuid() we defined above to create a uuid for each node based on its timestampe. And we'll use the helper functions create_date() and split_name() above to extract relevant metadata from the git log entry and add them to the node.

In [ ]

from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo


# Create a Node object from a single row of data
def create_node(row):
    record = row.to_dict()
    record_name = split_name(record["author"])
    record_content = (
        str(record["date"])
        + " "
        + record_name
        + " "
        + str(record["change summary"])
        + " "
        + str(record["change details"])
    )
    # Can change to TextNode as needed
    node = TextNode(
        id_=create_uuid(record["date"]),
        text=record_content,
        metadata={
            "commit": record["commit"],
            "author": record_name,
            "date": create_date(record["date"]),
        },
    )
    return node

from llama_index.core.schema import TextNode, NodeRelationship, RelatedNodeInfo # Create a Node object from a single row of data def create_node(row): record = row.to_dict() record_name = split_name(record["author"]) record_content = ( str(record["date"]) + " " + record_name + " " + str(record["change summary"]) + " " + str(record["change details"]) ) # Can change to TextNode as needed node = TextNode( id_=create_uuid(record["date"]), text=record_content, metadata={ "commit": record["commit"], "author": record_name, "date": create_date(record["date"]), }, ) return node

In [ ]

nodes = [create_node(row) for _, row in df.iterrows()]

nodes = [create_node(row) for _, row in df.iterrows()]

接下来，我们将为每个节点的内容创建向量嵌入，以便我们可以对与每个节点相关的文本执行相似性搜索。我们将使用 OpenAIEmbedding 模型创建嵌入。

In [ ]

# Create embeddings for nodes
from llama_index.embeddings.openai import OpenAIEmbedding

embedding_model = OpenAIEmbedding()

for node in nodes:
    node_embedding = embedding_model.get_text_embedding(
        node.get_content(metadata_mode="all")
    )
    node.embedding = node_embedding

# Create embeddings for nodes from llama_index.embeddings.openai import OpenAIEmbedding embedding_model = OpenAIEmbedding() for node in nodes: node_embedding = embedding_model.get_text_embedding( node.get_content(metadata_mode="all") ) node.embedding = node_embedding

让我们检查一下我们集合中的第一个节点，看看它长什么样。

In [ ]

print(nodes[0].get_content(metadata_mode="all"))

print(nodes[0].get_content(metadata_mode="all"))

commit: 44e41c12ab25e36c202f58e068ced262eadc8d16
author: Lakshmi Narayanan Sreethar
date: 2023-09-5 21:03:21+0850

Tue Sep 5 21:03:21 2023 +0530 Lakshmi Narayanan Sreethar Fix segfault in set_integer_now_func When an invalid function oid is passed to set_integer_now_func, it finds out that the function oid is invalid but before throwing the error, it calls ReleaseSysCache on an invalid tuple causing a segfault. Fixed that by removing the invalid call to ReleaseSysCache.  Fixes #6037

In [ ]

print(nodes[0].get_embedding())

print(nodes[0].get_embedding())

[-0.005366453900933266, 0.0016374519327655435, 0.005981510039418936, -0.026256779208779335, -0.03944991156458855, 0.026299940422177315, -0.0200558640062809, -0.01252412423491478, -0.04241368919610977, -0.004758591763675213, 0.05639812350273132, 0.006578581873327494, 0.014833281747996807, 0.009509989991784096, 0.0009675443288870156, -0.013157163746654987, -0.002265996066853404, -0.017048921436071396, 0.006553404498845339, -0.00217068032361567, 0.009085564874112606, 0.011775985360145569, -0.02514895796775818, -0.002679630182683468, 0.0030608929228037596, -3.439458305365406e-05, -0.00363818253390491, -0.03939236328005791, 0.0016806137282401323, -0.01207092497497797, 0.01739421673119068, -0.02241537719964981, -0.01753808930516243, -0.023782167583703995, -0.01598426327109337, -0.02575322426855564, -0.016876274719834328, -0.006380756851285696, -0.0009149408433586359, 0.00704616867005825, -0.0013290246715769172, -0.009776154533028603, -0.013200325891375542, -0.024832438677549362, -0.0019404839258641005, 0.027220726013183594, -0.004765785299241543, -0.008553235791623592, -0.023120352998375893, 0.006920279935002327, 0.017739512026309967, 0.0166892409324646, -0.019408436492085457, 0.010207772254943848, 0.01595548912882805, 0.004783769138157368, 0.008855368942022324, 0.018084805458784103, -0.012603254057466984, -0.002003428293392062, -0.0008407564600929618, 0.00394211383536458, -0.018948042765259743, 0.005722539033740759, -0.004244246520102024, -0.011502627283334732, -0.000936971337068826, 0.006873521022498608, -0.0038593867793679237, 0.0003349537728354335, 0.02490437589585781, 0.022861381992697716, -0.013833366334438324, 0.005657796282321215, 0.027896929532289505, -0.020415544509887695, -0.007143282797187567, 0.014862056821584702, -0.00667569600045681, -0.020199736580252647, 0.01827184110879898, -0.0030698850750923157, -0.032975636422634125, 0.02595464698970318, -0.0014818893978372216, -0.004906061105430126, 0.01008548028767109, 0.009337342344224453, -0.009833703748881817, -0.0011680669849738479, 0.010653777979314327, -0.0006110096583142877, 0.016228847205638885, -0.010589035227894783, 0.0010997274657711387, 0.020300446078181267, 0.005715345498174429, 0.009862477891147137, -0.0015664147213101387, -0.009207856841385365, -0.013480877503752708, -0.01759563945233822, 0.007992131635546684, -0.012639221735298634, -0.016833113506436348, -0.01654536835849285, 0.009366116486489773, 0.004229859448969364, -0.0044168937020003796, -0.00028122629737481475, -0.028918424621224403, 0.030616123229265213, -0.017020147293806076, -0.02500508539378643, 0.01844448782503605, 0.00011554780940059572, 0.021278781816363335, -0.01503470353782177, -0.024760503321886063, -0.02408429980278015, 0.03734936937689781, 0.000861438165884465, 0.021365106105804443, -0.006740438751876354, 0.005557085387408733, -0.017005760222673416, -0.01831500232219696, -0.01458150427788496, -0.0207896139472723, -0.004100373946130276, 0.011214882135391235, 0.03228504955768585, 0.00543119665235281, 0.02251608669757843, 0.011373141780495644, 0.0207896139472723, 0.004032033961266279, 0.019768116995692253, -0.016329558566212654, -0.02755163423717022, -0.0001643296709517017, 0.04163677617907524, -0.02163846418261528, 0.019394047558307648, -0.028975974768400192, 0.040543343871831894, 0.006010284647345543, 0.009812122210860252, 0.024746114388108253, -0.027781831100583076, -0.0009360721451230347, 0.002836091909557581, -0.008733076974749565, 0.010754489339888096, -0.005380841437727213, 0.01586916483938694, 0.0003014584071934223, 0.006862730719149113, -0.033666227012872696, -0.01664607785642147, 0.001758844475261867, 0.0125528983771801, 0.0065066455863416195, 0.016228847205638885, 0.010466743260622025, 0.0251057967543602, -0.009215050376951694, -0.016027426347136497, 0.0116033386439085, 0.019667407497763634, 0.008905723690986633, 0.011517014354467392, -0.0036561666056513786, 0.02263118512928486, 0.027868153527379036, 0.02509140968322754, 0.011984600685536861, -0.016473431140184402, -0.013847753405570984, 0.01377581711858511, -0.010315677151083946, -0.0076612248085439205, 0.031076516956090927, 0.06526068598031998, -0.01297012995928526, -0.008610785007476807, 0.02340809814631939, 0.0038989519234746695, 0.009251018986105919, 0.003494309727102518, 0.009301373735070229, 0.003737095044925809, -0.03757956624031067, -0.0013029477559030056, -0.5865404605865479, -0.013444909825921059, 0.0021041391883045435, -0.004449265077710152, 0.0020052266772836447, 0.010776069946587086, 0.025695675984025, 0.01172563061118126, -0.02909107320010662, 0.015250512398779392, -0.014487987384200096, 0.04822615161538124, 0.014487987384200096, -0.030357152223587036, -0.01828622817993164, -0.00993441417813301, 0.010294096544384956, 0.0014764942461624742, 0.005524714011698961, -0.001750751631334424, -0.020386770367622375, 0.010977491736412048, -0.01082642562687397, -0.010092674754559994, 0.021091746166348457, -0.009229437448084354, 0.012013375759124756, -0.01284064445644617, -0.00015736083150841296, 0.009063984267413616, -0.013660718686878681, 0.0058807991445064545, 0.013955658301711082, 0.0028414870612323284, 0.046960070729255676, -0.01130120549350977, -0.04943468049168587, 0.006970635149627924, 0.0046219127252697945, 0.031335487961769104, -0.031306713819503784, -0.026357490569353104, 0.042902857065200806, 0.01129401195794344, -0.010941523127257824, -0.0027857364621013403, 0.00627644918859005, -0.023005254566669464, -0.0133226178586483, -0.05228336155414581, 0.025494253262877464, 0.016084974631667137, 0.014329726807773113, 0.03809750825166702, -0.025393541902303696, -0.02322106435894966, 0.006686486769467592, 0.01847326196730137, 0.02989676035940647, 0.0017372636357322335, -0.018530812114477158, 0.0418669730424881, -0.010092674754559994, 0.003927726298570633, -0.009876864962279797, -0.012322702445089817, 0.007977744564414024, 0.027177564799785614, 0.036255937069654465, 0.007841065526008606, 0.02255924977362156, 0.02076083980500698, 0.00024390929320361465, -0.01254570484161377, -0.005535504315048456, 0.014185854233801365, 0.01211408618837595, -0.020199736580252647, -0.02414184994995594, 0.006977829150855541, -0.010157416574656963, -0.025997808203101158, 0.017235957086086273, -0.008862562477588654, 0.017077697440981865, 0.02084716409444809, 0.0029583836439996958, -0.003523084335029125, 0.007387866266071796, -0.0035122937988489866, 0.01923578791320324, 0.010092674754559994, -0.0016896057641133666, 0.0016887065721675754, 0.025666899979114532, -0.0024979908484965563, -0.01455992367118597, 0.009351729415357113, -0.013631944544613361, -0.03240014612674713, -0.020573804154992104, -0.029004748910665512, -0.02096226066350937, -0.013495265506207943, 0.0216672383248806, 0.038011182099580765, -0.01664607785642147, -0.03078877180814743, 0.037234269082546234, 0.0005889791063964367, -0.0040392279624938965, -0.012365863658487797, -0.009862477891147137, -0.011783178895711899, -0.008862562477588654, -0.02922055870294571, 0.025407928973436356, 0.001157276565209031, 0.008589203469455242, -0.007941776886582375, 0.005686570890247822, 0.007190041244029999, 0.01579722948372364, -0.02337932400405407, 0.007003006525337696, -0.00768999895080924, 0.028645066544413567, -0.019710568711161613, -0.021307555958628654, -0.0257676113396883, 0.012660803273320198, -0.006607356481254101, 0.010459549725055695, 0.0007481383509002626, 0.02342248521745205, -0.007632450200617313, 0.02916300855576992, -0.02093348652124405, 0.035565346479415894, -0.011891083791851997, -0.02571006305515766, -0.0050067720003426075, -0.006441902834922075, -0.01040919404476881, -0.001439626794308424, -0.011502627283334732, -0.03838525339961052, -0.004352150950580835, 0.01746615394949913, -0.00197825045324862, -0.008661140687763691, 0.004237052984535694, -0.041377805173397064, 0.01595548912882805, -0.003506898647174239, -0.004805350210517645, -0.010229353792965412, -0.016372719779610634, 0.005852024536579847, -0.007006603758782148, 0.007790710311383009, 0.02512018382549286, -0.01458150427788496, 0.020429931581020355, -0.006862730719149113, -0.006783600896596909, -0.009898446500301361, 0.00603905925527215, -0.015279287472367287, -0.03827015310525894, -0.009409278631210327, 0.0021796722430735826, -0.011941439472138882, -0.009330148808658123, -0.010286902077496052, 0.01004231907427311, -0.023667069151997566, -0.007948970422148705, -0.013502459041774273, 0.00689150532707572, 0.028832102194428444, 0.02832854725420475, -0.0332346074283123, -0.012416219338774681, 0.009891252033412457, 0.017192795872688293, 0.01844448782503605, 0.0008421052480116487, 0.013560008257627487, 0.025292832404375076, -0.023954814299941063, 0.009912833571434021, -0.003154410282149911, -0.01086239330470562, 0.011509820818901062, 0.03752201795578003, -0.004481636453419924, -0.009013628587126732, 0.004283811431378126, 0.030299603939056396, 0.014164273627102375, 0.006959844846278429, 0.02920617163181305, -0.011977407149970531, -0.0028288981411606073, -0.023796554654836655, 0.001507966429926455, -0.008546042256057262, -0.019796893000602722, -0.021998144686222076, -0.000644280225969851, -0.014718183316290379, -0.013085227459669113, -0.005549891851842403, 0.008733076974749565, 0.042068395763635635, 0.00501756276935339, 0.01585477776825428, -0.02152336575090885, -0.01127243135124445, -0.005700958427041769, -0.003776659956201911, 0.028947200626134872, 0.004992384929209948, 0.0016985977999866009, -0.008143198676407337, 0.004729817155748606, 0.016444656997919083, 0.022372214123606682, 0.0038773708511143923, 0.0027857364621013403, 0.012365863658487797, 0.02819906175136566, 0.01549509633332491, 0.04822615161538124, -0.010222159326076508, 0.01845887489616871, -0.012236378155648708, 0.03939236328005791, -0.003064489923417568, -0.015552645549178123, 0.014200241304934025, 0.02571006305515766, -0.009466827847063541, -0.024501532316207886, -0.02660207450389862, 0.03274543955922127, -0.0028936408925801516, 0.0067907944321632385, 0.026213617995381355, -0.007431028410792351, 0.0007823081687092781, -0.004006856586784124, 0.02745092287659645, 0.02737898752093315, 0.014315339736640453, 0.004862899426370859, 0.010315677151083946, 0.01989760249853134, 0.041982073336839676, 0.01996953971683979, 0.011092590168118477, 0.001743558095768094, -0.01131559256464243, 0.015236125327646732, 0.005107482895255089, 0.018027257174253464, -0.014962767250835896, 0.0006712563335895538, 0.01573967933654785, -0.02673156000673771, -0.022156406193971634, -0.04419771209359169, 0.00019040661572944373, 0.020458707585930824, 0.011258043348789215, 0.01904875412583351, 0.015164189040660858, 0.010517098940908909, -0.021998144686222076, -0.039593786001205444, -0.01992637850344181, 0.04448545724153519, 0.011452271603047848, -0.0019728553015738726, -0.021120522171258926, 0.0043809255585074425, 0.0013056453317403793, -0.017753899097442627, -0.0200558640062809, -0.019005591049790382, 0.02409868687391281, 0.014804507605731487, 0.006682890001684427, 0.01008548028767109, -0.00481614051386714, 0.030616123229265213, -0.003010537475347519, -0.014790119603276253, -0.006780004128813744, -0.028055189177393913, 0.017940932884812355, -0.01424340344965458, 0.0034511478152126074, 0.04736291244626045, -0.012301120907068253, -0.016502205282449722, -0.0018685475224629045, -0.009999156929552555, -0.004460055846720934, 0.03487475588917732, -0.009790541604161263, -0.019422823563218117, -0.03389642387628555, 0.009991963393986225, 0.0016770168440416455, -0.005945541895925999, -0.0014899822417646646, 0.04517604783177376, 0.008596397936344147, -0.021134909242391586, -0.03263034299015999, -0.025407928973436356, 0.012257959693670273, 0.026127293705940247, 0.05098850652575493, 0.004111164249479771, 0.021149296313524246, -0.021278781816363335, 0.00727636506780982, -0.013380167074501514, -0.014171467162668705, 0.03737814351916313, -0.019653018563985825, 0.018861718475818634, 0.012991710565984249, -0.0012795684160664678, -0.016358332708477974, 0.032112400978803635, 0.017120858654379845, 0.032054852694272995, -0.006474274210631847, -0.01131559256464243, -0.014667828567326069, -0.012689577415585518, 0.01907752826809883, 0.022012531757354736, 0.01740860380232334, 0.0033450417686253786, 0.02322106435894966, 0.03893196955323219, 0.033666227012872696, 0.025566190481185913, -0.017825834453105927, -0.02571006305515766, 0.011258043348789215, 0.023465648293495178, 0.012315508909523487, -0.027235113084316254, 0.020415544509887695, 0.005945541895925999, 0.0029961501713842154, 0.02083277516067028, -0.004118357785046101, 0.012236378155648708, -0.012941354885697365, 0.008776238188147545, 0.009661056101322174, 0.009078371338546276, 0.0009558546589687467, -0.005672183819115162, 0.008869756013154984, 0.0011509821051731706, 0.021883046254515648, 0.008085649460554123, -0.012135667726397514, -0.027896929532289505, -0.03406906872987747, 0.027824992313981056, 0.029724113643169403, -0.02762356959283352, -0.023034028708934784, -0.027321437373757362, -0.02746530994772911, -0.02409868687391281, -0.016372719779610634, -0.0012382050044834614, -0.005837637465447187, 0.001647343160584569, 0.016257621347904205, -0.01657414250075817, 0.0007503863889724016, -0.02595464698970318, 0.01595548912882805, -0.0030051423236727715, -0.002246213611215353, -0.02493315003812313, 0.015610194765031338, 0.008546042256057262, -0.0008038890664465725, 0.03818383067846298, -0.004553572740405798, 0.00098912522662431, -0.0023703037295490503, -0.01996953971683979, -0.042931631207466125, -0.02591148391366005, -0.03473088517785072, 0.010941523127257824, 0.0009603506769053638, 0.021307555958628654, -0.008920111693441868, -0.011142945848405361, 0.022688735276460648, 0.016444656997919083, -0.006197319366037846, 0.020660128444433212, 0.016516592353582382, -0.01410672441124916, 0.006949054542928934, 0.014682215638458729, 0.01007109321653843, 0.009200663305819035, 0.009488408453762531, 0.023623907938599586, 0.028573131188750267, -0.005967122968286276, -0.014675022102892399, -0.013185938820242882, 0.008323038928210735, -0.0018883299781009555, 0.010193385183811188, 0.006582179106771946, -0.0028288981411606073, 0.02163846418261528, -0.005456374492496252, 0.012783095240592957, 0.01589794084429741, 0.030501024797558784, 0.0026544525753706694, 0.035392697900533676, 0.012610447593033314, -0.0026112906634807587, -0.023810941725969315, 0.0035212859511375427, -0.01579722948372364, 0.03263034299015999, -0.025465479120612144, 0.006096608471125364, 0.009207856841385365, -0.009790541604161263, -0.02926371991634369, -0.003773063188418746, 0.023868491873145103, -0.01750931516289711, 0.016171298921108246, 0.033004410564899445, -0.00790580827742815, -0.00728355860337615, 0.0021077359560877085, -0.017955319955945015, -0.0032533227931708097, -0.0004282462759874761, -0.006204512901604176, 0.014430438168346882, -0.03985275700688362, 0.01591232791543007, -0.03519127890467644, -0.03749324008822441, -0.025379154831171036, -0.019552309066057205, -0.008078455924987793, 0.042874082922935486, 0.018502037972211838, -0.010545873083174229, -0.009085564874112606, -0.026069745421409607, -0.0023127547465264797, 0.00957473274320364, -0.02929249405860901, -0.03487475588917732, -0.01132997963577509, 0.03188220411539078, 0.009107145480811596, 0.030644899234175682, -0.013459296897053719, 0.04224104434251785, -0.016789952293038368, 0.0027731475420296192, 0.002381094265729189, 0.01130120549350977, 0.008934498764574528, -0.012337089516222477, 0.007517351768910885, 0.0005318796029314399, -0.005711748730391264, -0.01535122375935316, 0.0017957119271159172, -0.004132745321840048, 0.02592587098479271, -0.026285553351044655, -0.0012876612599939108, -0.01759563945233822, -0.029839210212230682, -0.003095062915235758, -0.0052477591671049595, -0.015567032620310783, 0.01664607785642147, -0.028098350390791893, 0.009984769858419895, 0.04307550564408302, 0.0010817432776093483, 0.00710731465369463, 0.016041813418269157, 0.010437969118356705, -0.028573131188750267, 0.010704133659601212, 0.005229774862527847, 0.002433248097077012, 0.012229184620082378, -0.0018793379422277212, -0.006312417332082987, 0.01743737794458866, -0.016948211938142776, 0.009502795524895191, 0.017782673239707947, -0.00690589239820838, -0.010553067550063133, -0.01595548912882805, 0.020228510722517967, -0.028558744117617607, -0.012186023406684399, 0.024443982169032097, -0.03150813654065132, 0.006003091111779213, -0.03320583328604698, -0.024659791961312294, -0.013876527547836304, 0.007312333211302757, 0.00689869886264205, 0.0004842217604164034, 0.020991036668419838, -0.007797903846949339, -0.014437631703913212, 0.0003288841398898512, -0.01674678921699524, 0.02819906175136566, -0.006826762575656176, 0.03763711452484131, 0.03332093358039856, 0.0006676595658063889, -0.046960070729255676, 0.01986882835626602, 0.03254402056336403, 0.005783685017377138, 0.013142776675522327, 0.02589709684252739, 0.009905640035867691, 0.006765616592019796, -0.00791300181299448, -2.810014905207936e-07, -0.03228504955768585, -0.018904881551861763, 0.01572529226541519, 0.02008463814854622, 0.014732571318745613, -0.009279793128371239, -0.011768791824579239, -0.021998144686222076, -0.011135751381516457, -0.01209969911724329, 0.005643409211188555, -0.008459718897938728, -0.0033558320719748735, -0.012826256453990936, 0.03844280168414116, -0.01502031646668911, 0.009776154533028603, 0.020458707585930824, -0.009704218246042728, -0.012373057194054127, -0.021839885041117668, -0.030587349086999893, -0.005729732569307089, -0.026199230924248695, 0.04943468049168587, 0.0026490571908652782, -0.02658768743276596, 0.002050186973065138, -0.010668165050446987, -0.016200073063373566, -0.046413354575634, -0.008589203469455242, 0.04224104434251785, -0.019336499273777008, -0.009661056101322174, 0.01995515264570713, 0.013509652577340603, -0.0022803833708167076, -0.005157838575541973, -0.016890661790966988, -0.006258465349674225, -0.03591063991189003, 0.003055497771129012, 0.010761682875454426, 0.004546379204839468, -0.003618400078266859, 0.009402085095643997, -0.016473431140184402, -0.018832944333553314, -0.015624581836163998, -0.035392697900533676, 0.009862477891147137, -0.04411138966679573, 0.01038761343806982, -0.030213279649615288, -0.01329384371638298, 0.030414702370762825, 0.02084716409444809, -0.03421294316649437, 0.013444909825921059, 0.04914693534374237, -0.031134065240621567, 0.009286986663937569, 0.00023739006428513676, 0.012811869382858276, -0.03507617861032486, -0.0007841065526008606, -0.020286059007048607, -0.013027678243815899, 0.025249669328331947, -0.009028015658259392, -0.023839715868234634, -0.017164019867777824, 0.02319229021668434, -0.0019171045860275626, 0.01906314119696617, 0.0008443532860837877, 0.05933312699198723, 0.012596060521900654, 0.01253131777048111, -0.034673336893320084, -0.004103970713913441, 0.03812628239393234, -0.02589709684252739, 0.0006658611237071455, 0.002077162964269519, -0.008301458321511745, 0.008596397936344147, -0.003589625470340252, -0.005057127680629492, 0.009358922950923443, -0.03746446594595909, 0.003701126901432872, -0.0028666649013757706, 0.00605344632640481, -0.023667069151997566, -0.016401495784521103, -0.021969370543956757, 0.007841065526008606, -0.0040859864093363285, 0.014703796245157719, -0.016243234276771545, -0.02001270093023777, -0.005798072554171085, 0.015279287472367287, -0.0018577571026980877, 0.011373141780495644, -0.01164649985730648, -0.0021329137962311506, -0.0023667069617658854, -0.025508640334010124, -0.0201277993619442, -0.007071346510201693, -0.01904875412583351, 0.03755079209804535, -0.011042234487831593, 0.005898783449083567, -0.011883890256285667, -0.023595133796334267, -0.02179672382771969, -0.00870430190116167, -0.02490437589585781, 0.011387528851628304, 0.02986798621714115, -0.0021958579309284687, -0.008265490643680096, 0.025235282257199287, 0.001056565553881228, 0.020688902586698532, -0.004470846150070429, 0.0018919268622994423, 0.015509483404457569, -0.004175907000899315, 0.025307219475507736, 0.012272346764802933, -0.019653018563985825, -0.02491876296699047, 0.004352150950580835, 0.029839210212230682, 0.041233934462070465, -0.011984600685536861, -0.0008515469380654395, -0.0076684183441102505, 0.0021868660114705563, -0.016099361702799797, -0.012682383880019188, -0.005197403486818075, 0.0012750723399221897, -0.018545199185609818, 0.0014576109824702144, 0.017681961879134178, 0.00019715064263436943, -0.024760503321886063, -0.017955319955945015, -0.011351561173796654, -0.019465984776616096, 0.009869671426713467, 0.005078708287328482, 0.010754489339888096, 0.024789277464151382, -0.02332177571952343, 0.0173510555177927, 0.0037838537245988846, -0.001575406757183373, 0.0241562370210886, -0.004233456216752529, 0.006970635149627924, -0.01333700492978096, 0.014653440564870834, -0.02986798621714115, 0.008517267182469368, 0.009200663305819035, -0.0011563773732632399, 0.026299940422177315, -0.008092842996120453, -0.01424340344965458, -0.018559586256742477, -0.02594025991857052, -0.009970382787287235, -0.0010502712102606893, -0.033666227012872696, -0.017739512026309967, -0.006596566177904606, -0.027868153527379036, -0.00879781972616911, -0.024458369240164757, -0.010178998112678528, -0.004575153812766075, -0.018948042765259743, -0.02156652696430683, -0.04434158653020859, 0.002605895511806011, 0.0007724168826825917, -0.0012507938081398606, -0.047880854457616806, -0.017825834453105927, -0.010164611041545868, 0.012804675847291946, 0.027249502018094063, 0.18795537948608398, 0.006362773012369871, 0.003310872009024024, 0.01674678921699524, -0.011970213614404202, -0.0014054570347070694, 0.008840980939567089, -0.010963104665279388, -0.037953633815050125, 0.04327692836523056, -0.011142945848405361, 0.010308483615517616, -0.008617978543043137, -0.0027335823979228735, -0.008085649460554123, 0.010301290079951286, -0.043823644518852234, 0.00564700597897172, -0.05435512959957123, -0.009905640035867691, 0.033608678728342056, -0.004006856586784124, -0.006643324624747038, -0.02178233675658703, 0.02596903406083584, -0.03418416902422905, 0.010704133659601212, 0.009761766530573368, 0.014085143804550171, 0.005226178094744682, -0.023710230365395546, 0.0055750696919858456, -0.000936971337068826, -0.026961755007505417, -0.01926456205546856, -0.037953633815050125, 0.012315508909523487, 0.008466912433505058, 0.017077697440981865, 0.0058843959122896194, 0.002050186973065138, -0.0251057967543602, 0.015178576111793518, -0.01746615394949913, -0.00251237815245986, 0.032831765711307526, -0.008956079371273518, 0.0013020484475418925, -0.012480962090194225, 0.03896074369549751, -0.04566521570086479, 0.009955994784832, 0.03170955553650856, 0.011121364310383797, 0.016789952293038368, -0.009452440775930882, -0.01335139200091362, -0.0038342091720551252, 0.022314665839076042, 0.009438052773475647, -0.023940427228808403, 0.033666227012872696, 0.0005417708889581263, 0.007790710311383009, -0.018861718475818634, 0.011696855537593365, -0.02926371991634369, 0.005657796282321215, 0.02402675151824951, -0.04578031226992607, 0.009509989991784096, -0.024875599890947342, 0.004308989271521568, -0.007182847708463669, -0.013200325891375542, -0.00787703413516283, 0.0005876303184777498, 0.00913592055439949, 0.009840897284448147, 0.014437631703913212, -0.0050103687681257725, -0.04721904173493385, -0.007344704587012529, -0.0003257369389757514, -0.042010847479104996, -0.03510495275259018, 0.028832102194428444, 0.022314665839076042, -0.00880501326173544, -0.006744035519659519, 0.0007535336189903319, 0.011351561173796654, 0.011847921647131443, -0.005021159537136555, 0.005154241807758808, 0.014286565594375134, -0.00790580827742815, 0.00566139305010438, -0.022976480424404144, 0.004111164249479771, -0.013502459041774273, 0.05501694604754448, -0.011704049073159695, -0.044600557535886765, -0.00791300181299448, 0.009646669030189514, 0.012236378155648708, 0.006909489631652832, 0.010596228763461113, -0.027954477816820145, -0.00710731465369463, -0.0499238483607769, -0.011991795152425766, -0.007524545304477215, -0.00540961604565382, 0.025307219475507736, 0.0064634839072823524, -0.033004410564899445, 0.002731784014031291, -0.013193132355809212, 0.02581077441573143, -0.0014908815501257777, 0.02258802391588688, 0.018775396049022675, -0.0019998312927782536, 0.0010511704022064805, -0.010790457017719746, -0.008517267182469368, -0.00603186571970582, -0.04321937635540962, 0.015653356909751892, -0.0073806727305054665, 0.010452356189489365, -0.013099615462124348, -0.007869839668273926, -0.00067800038959831, 0.017279118299484253, 6.013431993778795e-05, 0.007096523884683847, -0.008653946220874786, -0.003602214390411973, -0.016833113506436348, -0.011797566898167133, -0.011344367638230324, 0.0015385393053293228, -0.005121870432049036, 0.023868491873145103, 0.0026652428787201643, -0.021422654390335083, 0.007395059801638126, -0.019336499273777008, -0.004528395365923643, 0.014272177591919899, 0.009783348068594933, 0.004319779574871063, -0.010617810301482677, -0.030443476513028145, -0.041406579315662384, 0.005366453900933266, -0.007157669868320227, -0.02171039953827858, 0.018919268622994423, 0.037780988961458206, 0.0038342091720551252, -0.03165200725197792, 0.003003343939781189, -0.18473263084888458, -0.0065066455863416195, -0.018631523475050926, -0.034845981746912, 0.027120016515254974, -0.011761598289012909, 0.031018966808915138, 0.0005489644827321172, -0.006830359343439341, -0.01999831385910511, 0.011416303925216198, 0.0192501749843359, -0.049578554928302765, -0.014660634100437164, -0.004258633591234684, -0.015192964114248753, 0.014732571318745613, -0.0006928372895345092, 0.03691774979233742, 0.017739512026309967, 0.018214290961623192, -0.0349898561835289, 0.010553067550063133, 0.009215050376951694, 0.00868272129446268, 0.03240014612674713, -0.004478039685636759, -0.017897771671414375, -0.009438052773475647, -0.014962767250835896, -0.003569843014702201, 0.018530812114477158, 0.02924933284521103, 0.0018083007307723165, 0.0029098265804350376, -0.0023361339699476957, -0.007431028410792351, -3.0151459213811904e-05, 0.015178576111793518, -0.004916851874440908, 0.018185516819357872, 0.018142355605959892, -0.0067943911999464035, 0.026472589001059532, -0.01988321542739868, 0.017926545813679695, 0.04998139664530754, -0.0366012305021286, 0.01128681842237711, -0.021106135100126266, 0.02832854725420475, -0.025278443470597267, -0.009682636708021164, 0.004154325928539038, 0.021897435188293457, 0.005571472924202681, 0.01586916483938694, 0.021868659183382988, -0.03099019266664982, -6.738415686413646e-05, -0.0005327787948772311, -0.018631523475050926, 0.015293674543499947, -0.00353387463837862, 0.0028648662846535444, -0.004891674034297466, -0.010668165050446987, -0.00961789395660162, -0.00768999895080924, 0.017797060310840607, -0.020631354302167892, -0.011660887859761715, 0.005085902288556099, 0.012171635404229164, 0.008121617138385773, 0.028932811692357063, -0.031076516956090927, 0.009229437448084354, 0.003350436920300126, -0.005826846696436405, -0.004902464337646961, 0.042989183217287064, -0.009891252033412457, -0.008855368942022324, -0.00239368318580091, -0.01040200050920248, -0.010660971514880657, -0.012495349161326885, 0.008071262389421463, -0.02008463814854622, 0.01421462930738926, -0.018228679895401, -0.014862056821584702, -0.011768791824579239, -0.01502031646668911, 0.004028437193483114, -0.0033971955999732018, -0.008164779283106327, 0.01254570484161377, -0.008006519638001919, -0.016789952293038368, -0.020314833149313927, -0.003051901003345847, -0.003602214390411973, -0.012747126631438732, 0.0015700114890933037, -0.03668755292892456, -0.0027929299976676702, 0.020616967231035233, 0.011761598289012909, -0.011222075670957565, 0.03196852654218674, 0.009876864962279797, 0.02583954855799675, -0.009862477891147137, 0.0027821394614875317, -0.0038665805477648973, 0.006823165807873011, 0.017063310369849205, -0.020199736580252647, 0.041406579315662384, -0.0116033386439085, -0.011761598289012909, 0.023494422435760498, -0.011876696720719337, -0.023839715868234634, -0.09668249636888504, -0.031076516956090927, -0.00787703413516283, 0.022789444774389267, -0.014164273627102375, 0.030242053791880608, 0.003294686321169138, 0.033579904586076736, -0.007956163957715034, 0.0083446204662323, -0.0047190263867378235, -0.04445668309926987, -0.0008317644242197275, 0.012351476587355137, -0.0026076938956975937, 0.011027847416698933, -0.02244415134191513, -0.017048921436071396, 0.0200558640062809, 0.03429926559329033, 0.02419939823448658, -0.021839885041117668, 0.011970213614404202, -0.013207519426941872, -0.028012026101350784, -0.003449349431321025, -0.017782673239707947, 0.0173510555177927, 0.005963526200503111, 0.019523533061146736, 0.0380687341094017, 0.003035715315490961, 0.04742046073079109, -0.012588866986334324, -0.03691774979233742, -0.027925703674554825, -0.018199903890490532, 0.003346840152516961, 0.02432888373732567, -0.042845308780670166, 0.01912068948149681, 0.00871868897229433, 0.009286986663937569, 0.005618231371045113, -0.023091578856110573, 0.005111079663038254, -0.02248731255531311, 0.03752201795578003, 0.04575153812766075, -0.01504909060895443, -0.0402555987238884, 0.02737898752093315, -0.00015803524001967162, -0.021019810810685158, 0.003553657326847315, -0.007510158233344555, 0.03435681387782097, 0.006114592310041189, -0.00917188823223114, -0.007315929979085922, 0.008136005140841007, -0.0019332902738824487, 0.015192964114248753, 0.009653862565755844, -0.026961755007505417, 0.009898446500301361, 0.0005602045566774905, -0.03196852654218674, 0.03087509423494339, -0.01756686344742775, -0.017048921436071396, 0.022329052910208702, -0.025666899979114532, 0.009416472166776657, -0.01910630241036415, 0.0036201984621584415, -1.7815495084505528e-05, -0.02760918252170086, 0.0330907367169857, -0.010898361913859844, -0.021192457526922226, -0.018084805458784103, -0.021437041461467743, 0.01674678921699524, 0.0108336191624403, 0.014077950268983841, 0.009445247240364552, 0.021264394745230675, 0.04825492575764656, -0.018746620044112206, -0.0029637787956744432, 0.0055462950840592384, 0.004226262215524912, -0.004301795735955238, 0.010977491736412048, 0.03979520499706268, 0.00434855418279767, 0.014876443892717361, 0.012466575019061565, -0.01500592939555645, -0.05447022616863251, 0.0012391041964292526, -0.08776238560676575, 0.04744923859834671, 0.011761598289012909, 0.0018631522543728352, 0.0025213700719177723, 0.008898530155420303, 0.0028504792135208845, -0.029436366632580757, -0.013416134752333164, 0.023710230365395546, -0.019796893000602722, 0.02408429980278015, 0.0043737320229411125, -0.000470733706606552, -0.0023667069617658854, -0.02506263554096222, 0.026904206722974777, 0.011704049073159695, 0.03904706612229347, -0.0019566696137189865, -0.0036148030776530504, -0.015538258478045464, -0.04005417600274086, 0.032227497547864914, 0.00518661318346858, -0.0316232331097126, 0.006064237095415592, -0.001987242605537176, 0.00768280541524291, -0.014387276023626328, 0.009718605317175388, -0.045435018837451935, 0.014185854233801365, 0.02838609553873539, 0.0023325372021645308, -0.02509140968322754, 0.005607441067695618, 0.0038342091720551252, 0.037867311388254166, 0.008934498764574528, -0.019537921994924545, -0.015192964114248753, -0.008848174475133419, 0.003163402434438467, 0.01165369339287281, -0.013092420995235443, -0.02582516148686409, 0.009768960997462273, 0.013660718686878681, 0.003801837796345353, 0.024846825748682022, 0.009028015658259392, -0.02255924977362156, 0.0010457751341164112, -0.0034835191909223795, -0.01376143004745245, 0.005657796282321215, 0.01582600362598896, -0.012459381483495235, -0.027911316603422165, 0.037032850086688995, -0.012905387207865715, 0.021293168887495995, -0.027767444029450417, 0.013711074367165565, 0.0028217046055942774, 0.00022333998640533537, 0.004111164249479771, 0.02421378530561924, 0.02842925861477852, -0.020257284864783287, -0.013689493760466576, 0.02169601246714592, 0.008215134963393211, 1.9108101696474478e-05, 0.010934329591691494, -0.018674684688448906, -0.006970635149627924, -0.010596228763461113, 0.0012220193166285753, 0.017868997529149055, -0.03596819192171097, -0.013833366334438324, -0.021422654390335083, 0.01211408618837595, -0.010437969118356705, 0.003111248603090644, -0.0020879535004496574, -0.014905218034982681, -0.013020484708249569, -0.03919094055891037, 0.010999072343111038, -0.012452187947928905, -0.0074238344095647335, 0.004161519464105368, 0.016228847205638885, 0.0009432657971046865, 0.013847753405570984, 0.006326804868876934, -0.004686655011028051, 0.006736841984093189, -0.004014050122350454, -0.02337932400405407, -0.025494253262877464, -0.022918930277228355, -0.007625256199389696, -0.019782504066824913, -0.020616967231035233, -0.007546126376837492, -0.008840980939567089, 0.022213954478502274, 0.01412830501794815, -0.003382808296009898, 0.026184841990470886, -0.021278781816363335, 0.0027929299976676702, 0.022199567407369614, -0.00439531309530139, -0.004470846150070429, 0.023882878944277763, 0.0055966502986848354, 0.01752370223402977, 0.03401152044534683, 0.007524545304477215, 0.023739006370306015, 0.011214882135391235, 0.029666563495993614, -0.02242976427078247, 0.0077187735587358475, -0.030673673376441002, 0.011869503185153008, -0.005143451038748026, -0.013545620255172253, -0.012624834664165974, -0.02844364568591118, -0.0009243824752047658, -0.037867311388254166, 0.020214123651385307, -0.005165032111108303, 0.07366285473108292, 0.0038521932438015938, -0.020559417083859444, -0.03855790197849274, 0.018070418387651443, -0.007218815851956606, 0.01831500232219696, -0.0014189451467245817, 0.028760164976119995, -0.027666732668876648, 0.022890156134963036, 0.0019530727295204997, 0.011862309649586678, 0.0022030516993254423, -0.012243571691215038, -0.008661140687763691, 0.0015169584657996893, 0.02845803275704384, -0.010711327195167542, 0.007319526746869087, 0.022170793265104294, 0.009186276234686375, 0.019422823563218117, 0.018948042765259743, 0.0028810519725084305, -0.005330485757440329, 0.03004063293337822, -0.002987158251926303, -0.0058700088411569595, -0.04316182807087898, -0.0008155787363648415, -0.012286733835935593, 0.011552982963621616, -0.00666850246489048, 0.0076612248085439205, -0.022026920691132545, -0.001176159828901291, 0.0009864276507869363, 0.00605344632640481, 0.010013544000685215, -0.011912664398550987, 0.010466743260622025, -0.0481685996055603, -0.020458707585930824, 0.016818726435303688, -0.02411307580769062, 0.017912158742547035, 0.002213842235505581, -0.022026920691132545]

将文档和元数据加载到 TimescaleVector 向量存储中¶

现在我们已经准备好节点并将嵌入添加到其中，让我们将它们添加到 TimescaleVector 向量存储中。

我们将从创建的节点列表中创建一个 Timescale Vector 实例。

首先，我们将定义一个集合名称，它将作为我们在 PostgreSQL 数据库中的表名。

我们还将定义一个时间差（time delta），并将其作为 time_partition_interval 参数传递，该参数将用作按时间划分数据的间隔。每个分区将包含指定时长的数据。为了简单起见，我们将使用 7 天，但您可以根据您的用例选择任何合理的值——例如，如果您经常查询最近的向量，您可能希望使用更小的时间差，如 1 天，或者如果您查询跨越十年时间段的向量，您可能希望使用更大的时间差，如 6 个月或 1 年。

然后我们将节点添加到 Timescale Vector 向量存储中。

In [ ]

# Create a timescale vector store and add the newly created nodes to it
ts_vector_store = TimescaleVectorStore.from_params(
    service_url=TIMESCALE_SERVICE_URL,
    table_name="li_commit_history",
    time_partition_interval=timedelta(days=7),
)
_ = ts_vector_store.add(nodes)

# Create a timescale vector store and add the newly created nodes to it ts_vector_store = TimescaleVectorStore.from_params( service_url=TIMESCALE_SERVICE_URL, table_name="li_commit_history", time_partition_interval=timedelta(days=7), ) _ = ts_vector_store.add(nodes)

按时间和相似性查询向量¶

现在我们已经将文档加载到 TimescaleVector 中，我们可以按时间和相似性查询它们。

TimescaleVector 提供了多种方法，通过基于时间的过滤进行相似性搜索来查询向量。让我们看看下面的每种方法。

首先我们定义一个查询字符串并获取该查询字符串的向量嵌入。

In [ ]

# Define query and generate embedding for it
query_str = "What's new with TimescaleDB functions?"
embed_model = OpenAIEmbedding()
query_embedding = embed_model.get_query_embedding(query_str)

# Define query and generate embedding for it query_str = "What's new with TimescaleDB functions?" embed_model = OpenAIEmbedding() query_embedding = embed_model.get_query_embedding(query_str)

然后我们设置一些将在时间过滤器中使用的变量。

In [ ]

# Time filter variables for query
start_dt = datetime(
    2023, 8, 1, 22, 10, 35
)  # Start date = 1 August 2023, 22:10:35
end_dt = datetime(
    2023, 8, 30, 22, 10, 35
)  # End date = 30 August 2023, 22:10:35
td = timedelta(days=7)  # Time delta = 7 days

# Time filter variables for query start_dt = datetime( 2023, 8, 1, 22, 10, 35 ) # Start date = 1 August 2023, 22:10:35 end_dt = datetime( 2023, 8, 30, 22, 10, 35 ) # End date = 30 August 2023, 22:10:35 td = timedelta(days=7) # Time delta = 7 days

方法 1：在提供的开始日期和结束日期范围内过滤。

In [ ]

# Query the vector database
vector_store_query = VectorStoreQuery(
    query_embedding=query_embedding, similarity_top_k=5
)

# return most similar vectors to query between start date and end date date range
# returns a VectorStoreQueryResult object
query_result = ts_vector_store.query(
    vector_store_query, start_date=start_dt, end_date=end_dt
)
query_result

# Query the vector database vector_store_query = VectorStoreQuery( query_embedding=query_embedding, similarity_top_k=5 ) # return most similar vectors to query between start date and end date date range # returns a VectorStoreQueryResult object query_result = ts_vector_store.query( vector_store_query, start_date=start_dt, end_date=end_dt ) query_result

Out[ ]

VectorStoreQueryResult(nodes=[TextNode(id_='22747180-31f1-11ee-bd8e-101e36c28c91', embedding=None, metadata={'commit': ' 7aeed663b9c0f337b530fd6cad47704a51a9b2ec', 'author': 'Dmitry Simonenko', 'date': '2023-08-3 14:30:23+0500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='3273f20a98f02c75847896b929888b05e8751ae5e258d7feb8605bd5290ef8ca', text='Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='faa8ea00-4686-11ee-b933-c2c7df407c25', embedding=None, metadata={'commit': ' e4facda540286b0affba47ccc63959fefe2a7b26', 'author': 'Sven Klemm', 'date': '2023-08-29 18:13:24+0320'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='6f45ab1cccf673ddf75c625983b6cf2f4a66bbf865a4c1c65025997a470f3bb3', text='Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were moved into the _timescaledb_functions schema to improve schema security. This patch adds a compatibility layer so external callers of these internal functions will not break and allow for more flexibility when migrating. ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='d7080180-40d2-11ee-af6f-f43e81a0925a', embedding=None, metadata={'commit': ' cf04496e4b4237440274eb25e4e02472fc4e06fc', 'author': 'Sven Klemm', 'date': '2023-08-22 12:01:19+0320'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='d5a20dc83ae04f44aa901ba2f654e80ca68cb21f6a313bd91afcd91e404b471e', text='Tue Aug 22 12:01:19 2023 +0200 Sven Klemm Move utility functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - generate_uuid() - get_git_commit() - get_os_info() - tsl_loaded() ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='01b10780-4649-11ee-a375-5719b2881af3', embedding=None, metadata={'commit': ' a9751ccd5eb030026d7b975d22753f5964972389', 'author': 'Sven Klemm', 'date': '2023-08-29 10:49:47+0320'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='8fde14d147def41808d82bf2ffa35e1e0ed78b0331962907cee856af34a34e44', text='Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - get_partition_for_key(val anyelement) - get_partition_hash(val anyelement) ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='e7ba7f80-36af-11ee-9479-6c18a6a65db1', embedding=None, metadata={'commit': ' 44eab9cf9bef34274c88efd37a750eaa74cd8044', 'author': 'Konstantina Skovola', 'date': '2023-08-9 15:26:03+0500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='f0db9c719928ecc16653bbf0a44f1eaeb221ac79dae16fd36710044d1561dbfa', text='Wed Aug 9 15:26:03 2023 +0300 Konstantina Skovola Release 2.11.2 This release contains bug fixes since the 2.11.1 release. We recommend that you upgrade at the next available opportunity.  **Features** * #5923 Feature flags for TimescaleDB features  **Bugfixes** * #5680 Fix DISTINCT query with JOIN on multiple segmentby columns * #5774 Fixed two bugs in decompression sorted merge code * #5786 Ensure pg_config --cppflags are passed * #5906 Fix quoting owners in sql scripts. * #5912 Fix crash in 1-step integer policy creation  **Thanks** * @mrksngl for submitting a PR to fix extension upgrade scripts * @ericdevries for reporting an issue with DISTINCT queries using segmentby columns of compressed hypertable ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n')], similarities=[0.18141598590707553, 0.1821951378700205, 0.1948705199438009, 0.19657938500765504, 0.19664154042725346], ids=[UUID('22747180-31f1-11ee-bd8e-101e36c28c91'), UUID('faa8ea00-4686-11ee-b933-c2c7df407c25'), UUID('d7080180-40d2-11ee-af6f-f43e81a0925a'), UUID('01b10780-4649-11ee-a375-5719b2881af3'), UUID('e7ba7f80-36af-11ee-9479-6c18a6a65db1')])

让我们检查从相似性搜索返回的节点

In [ ]

# for each node in the query result, print the node metadata date
for node in query_result.nodes:
    print("-" * 80)
    print(node.metadata["date"])
    print(node.get_content(metadata_mode="all"))

# for each node in the query result, print the node metadata date for node in query_result.nodes: print("-" * 80) print(node.metadata["date"]) print(node.get_content(metadata_mode="all"))

--------------------------------------------------------------------------------
2023-08-3 14:30:23+0500
commit:  7aeed663b9c0f337b530fd6cad47704a51a9b2ec
author: Dmitry Simonenko
date: 2023-08-3 14:30:23+0500

Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create
--------------------------------------------------------------------------------
2023-08-29 18:13:24+0320
commit:  e4facda540286b0affba47ccc63959fefe2a7b26
author: Sven Klemm
date: 2023-08-29 18:13:24+0320

Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were moved into the _timescaledb_functions schema to improve schema security. This patch adds a compatibility layer so external callers of these internal functions will not break and allow for more flexibility when migrating.
--------------------------------------------------------------------------------
2023-08-22 12:01:19+0320
commit:  cf04496e4b4237440274eb25e4e02472fc4e06fc
author: Sven Klemm
date: 2023-08-22 12:01:19+0320

Tue Aug 22 12:01:19 2023 +0200 Sven Klemm Move utility functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - generate_uuid() - get_git_commit() - get_os_info() - tsl_loaded()
--------------------------------------------------------------------------------
2023-08-29 10:49:47+0320
commit:  a9751ccd5eb030026d7b975d22753f5964972389
author: Sven Klemm
date: 2023-08-29 10:49:47+0320

Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - get_partition_for_key(val anyelement) - get_partition_hash(val anyelement)
--------------------------------------------------------------------------------
2023-08-9 15:26:03+0500
commit:  44eab9cf9bef34274c88efd37a750eaa74cd8044
author: Konstantina Skovola
date: 2023-08-9 15:26:03+0500

Wed Aug 9 15:26:03 2023 +0300 Konstantina Skovola Release 2.11.2 This release contains bug fixes since the 2.11.1 release. We recommend that you upgrade at the next available opportunity.  **Features** * #5923 Feature flags for TimescaleDB features  **Bugfixes** * #5680 Fix DISTINCT query with JOIN on multiple segmentby columns * #5774 Fixed two bugs in decompression sorted merge code * #5786 Ensure pg_config --cppflags are passed * #5906 Fix quoting owners in sql scripts. * #5912 Fix crash in 1-step integer policy creation  **Thanks** * @mrksngl for submitting a PR to fix extension upgrade scripts * @ericdevries for reporting an issue with DISTINCT queries using segmentby columns of compressed hypertable

注意查询如何只返回指定日期范围内的结果。

方法 2：在提供的开始日期及之后的时间差范围内过滤。

In [ ]

vector_store_query = VectorStoreQuery(
    query_embedding=query_embedding, similarity_top_k=5
)

# return most similar vectors to query from start date and a time delta later
query_result = ts_vector_store.query(
    vector_store_query, start_date=start_dt, time_delta=td
)

for node in query_result.nodes:
    print("-" * 80)
    print(node.metadata["date"])
    print(node.get_content(metadata_mode="all"))

vector_store_query = VectorStoreQuery( query_embedding=query_embedding, similarity_top_k=5 ) # return most similar vectors to query from start date and a time delta later query_result = ts_vector_store.query( vector_store_query, start_date=start_dt, time_delta=td ) for node in query_result.nodes: print("-" * 80) print(node.metadata["date"]) print(node.get_content(metadata_mode="all"))

--------------------------------------------------------------------------------
2023-08-3 14:30:23+0500
commit:  7aeed663b9c0f337b530fd6cad47704a51a9b2ec
author: Dmitry Simonenko
date: 2023-08-3 14:30:23+0500

Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create
--------------------------------------------------------------------------------
2023-08-7 19:49:47+-500
commit:  5bba74a2ec083728f8e93e09d03d102568fd72b5
author: Fabrízio de Royes Mello
date: 2023-08-7 19:49:47+-500

Mon Aug 7 19:49:47 2023 -0300 Fabrízio de Royes Mello Relax strong table lock when refreshing a CAGG When refreshing a Continuous Aggregate we take a table lock on _timescaledb_catalog.continuous_aggs_invalidation_threshold when processing the invalidation logs (the first transaction of the refresh Continuous Aggregate procedure). It means that even two different Continuous Aggregates over two different hypertables will wait each other in the first phase of the refreshing procedure. Also it lead to problems when a pg_dump is running because it take an AccessShareLock on tables so Continuous Aggregate refresh execution will wait until the pg_dump finish.  Improved it by relaxing the strong table-level lock to a row-level lock so now the Continuous Aggregate refresh procedure can be executed in multiple sessions with less locks.  Fix #3554
--------------------------------------------------------------------------------
2023-08-3 14:36:39+0500
commit:  2863daf3df83c63ee36c0cf7b66c522da5b4e127
author: Dmitry Simonenko
date: 2023-08-3 14:36:39+0500

Thu Aug 3 14:36:39 2023 +0300 Dmitry Simonenko Support CREATE INDEX ONLY ON main table This PR adds support for CREATE INDEX ONLY ON clause which allows to create index only on the main table excluding chunks.  Fix #5908
--------------------------------------------------------------------------------
2023-08-2 20:24:14+0140
commit:  3af0d282ea71d9a8f27159a6171e9516e62ec9cb
author: Lakshmi Narayanan Sreethar
date: 2023-08-2 20:24:14+0140

Wed Aug 2 20:24:14 2023 +0100 Lakshmi Narayanan Sreethar PG16: ExecInsertIndexTuples requires additional parameter PG16 adds a new boolean parameter to the ExecInsertIndexTuples function to denote if the index is a BRIN index, which is then used to determine if the index update can be skipped. The fix also removes the INDEX_ATTR_BITMAP_ALL enum value.  Adapt these changes by updating the compat function to accomodate the new parameter added to the ExecInsertIndexTuples function and using an alternative for the removed INDEX_ATTR_BITMAP_ALL enum value.  postgres/postgres@19d8e23
--------------------------------------------------------------------------------
2023-08-7 16:36:17+0500
commit:  373c55662ca5f8a2993abf9b2aa7f5f4006b3229
author: Konstantina Skovola
date: 2023-08-7 16:36:17+0500

Mon Aug 7 16:36:17 2023 +0300 Konstantina Skovola Fix ordered append for partially compressed chunks In the exclusive presence of partially compressed chunks, this optimization was not applied because no pathkeys were supplied. Additionally, this patch makes sure that if applicable, the `enable_decompression_sorted_merge` optimization is chosen for the path, since it is more beneficial due to the ability to push down the sort below DecompressChunk.

再次注意，只返回开始日期（8 月 1 日）和定义的时间差（7 天后）之间的节点。

方法 3：在提供的结束日期及之前的时间差范围内过滤。

In [ ]

vector_store_query = VectorStoreQuery(
    query_embedding=query_embedding, similarity_top_k=5
)

# return most similar vectors to query from end date and a time delta earlier
query_result = ts_vector_store.query(
    vector_store_query, end_date=end_dt, time_delta=td
)

for node in query_result.nodes:
    print("-" * 80)
    print(node.metadata["date"])
    print(node.get_content(metadata_mode="all"))

vector_store_query = VectorStoreQuery( query_embedding=query_embedding, similarity_top_k=5 ) # return most similar vectors to query from end date and a time delta earlier query_result = ts_vector_store.query( vector_store_query, end_date=end_dt, time_delta=td ) for node in query_result.nodes: print("-" * 80) print(node.metadata["date"]) print(node.get_content(metadata_mode="all"))

--------------------------------------------------------------------------------
2023-08-29 18:13:24+0320
commit:  e4facda540286b0affba47ccc63959fefe2a7b26
author: Sven Klemm
date: 2023-08-29 18:13:24+0320

Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were moved into the _timescaledb_functions schema to improve schema security. This patch adds a compatibility layer so external callers of these internal functions will not break and allow for more flexibility when migrating.
--------------------------------------------------------------------------------
2023-08-29 10:49:47+0320
commit:  a9751ccd5eb030026d7b975d22753f5964972389
author: Sven Klemm
date: 2023-08-29 10:49:47+0320

Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - get_partition_for_key(val anyelement) - get_partition_hash(val anyelement)
--------------------------------------------------------------------------------
2023-08-28 23:26:23+0320
commit:  b2a91494a11d8b82849b6f11f9ea6dc26ef8a8cb
author: Sven Klemm
date: 2023-08-28 23:26:23+0320

Mon Aug 28 23:26:23 2023 +0200 Sven Klemm Move ddl_internal functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - chunk_constraint_add_table_constraint(_timescaledb_catalog.chunk_constraint) - chunk_drop_replica(regclass,name) - chunk_index_clone(oid) - chunk_index_replace(oid,oid) - create_chunk_replica_table(regclass,name) - drop_stale_chunks(name,integer[]) - health() - hypertable_constraint_add_table_fk_constraint(name,name,name,integer) - process_ddl_event() - wait_subscription_sync(name,name,integer,numeric)
--------------------------------------------------------------------------------
2023-08-29 14:47:57+0320
commit:  08231c8aacd17152f315ad36d95c031fb46073aa
author: Jan Nidzwetzki
date: 2023-08-29 14:47:57+0320

Tue Aug 29 14:47:57 2023 +0200 Jan Nidzwetzki Export is_decompress_chunk_path / is_gapfill_path This patch adds the 'ts_' prefix to the function names of is_decompress_chunk_path and is_gapfill_path and makes them available for use by other parts of TimescaleDB.
--------------------------------------------------------------------------------
2023-08-28 15:32:54+0320
commit:  6576d969b319dac8e7fd08a9cf4cfc8197b34d1d
author: Sven Klemm
date: 2023-08-28 15:32:54+0320

Mon Aug 28 15:32:54 2023 +0200 Sven Klemm Move log invalidation functions to _timescaledb_functions schema To increase schema security we do not want to mix our own internal objects with user objects. Since chunks are created in the _timescaledb_internal schema our internal functions should live in a different dedicated schema. This patch make the necessary adjustments for the following functions:  - cagg_watermark(integer) - cagg_watermark_materialized(integer) - hypertable_invalidation_log_delete(integer) - invalidation_cagg_log_add_entry(integer,bigint,bigint) - invalidation_hyper_log_add_entry(integer,bigint,bigint) - invalidation_process_cagg_log(integer,integer,regtype,bigint,bigint,integer[],bigint[],bigint[]) - invalidation_process_cagg_log(integer,integer,regtype,bigint,bigint,integer[],bigint[],bigint[],text[]) - invalidation_process_hypertable_log(integer,integer,regtype,integer[],bigint[],bigint[]) - invalidation_process_hypertable_log(integer,integer,regtype,integer[],bigint[],bigint[],text[]) - materialization_invalidation_log_delete(integer)

主要的重点是，在上面的每个结果中，只返回指定时间范围内的向量。这些查询非常高效，因为它们只需要搜索相关的分区。

4. 将 TimescaleVector 存储用作检索器和查询引擎¶

现在我们已经探索了基本的相似性搜索和带时间过滤器的相似性搜索，接下来让我们看看如何将 Timescale Vector 的这些特性与 LLamaIndex 的检索器和查询引擎结合使用。

首先，我们将介绍如何使用 TimescaleVector 作为[检索器](https://gpt-index.readthedocs.io/en/stable/api_reference/query/retrievers.html)，特别是[向量存储检索器](https://gpt-index.readthedocs.io/en/stable/api_reference/query/retrievers/vector_store.html)。

为了将检索到的节点限制在相关的时间范围内，我们可以使用 TimescaleVector 的时间过滤器。在创建检索器时，只需将时间过滤器参数作为 vector_strored_kwargs 传递即可。

In [ ]

from llama_index.core import VectorStoreIndex
from llama_index.core import StorageContext

index = VectorStoreIndex.from_vector_store(ts_vector_store)
retriever = index.as_retriever(
    vector_store_kwargs=({"start_date": start_dt, "time_delta": td})
)
retriever.retrieve("What's new with TimescaleDB functions?")

from llama_index.core import VectorStoreIndex from llama_index.core import StorageContext index = VectorStoreIndex.from_vector_store(ts_vector_store) retriever = index.as_retriever( vector_store_kwargs=({"start_date": start_dt, "time_delta": td}) ) retriever.retrieve("What's new with TimescaleDB functions?")

Out[ ]

[NodeWithScore(node=TextNode(id_='22747180-31f1-11ee-bd8e-101e36c28c91', embedding=None, metadata={'commit': ' 7aeed663b9c0f337b530fd6cad47704a51a9b2ec', 'author': 'Dmitry Simonenko', 'date': '2023-08-3 14:30:23+0500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='3273f20a98f02c75847896b929888b05e8751ae5e258d7feb8605bd5290ef8ca', text='Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds several GUCs which allow to enable/disable major timescaledb features:  - enable_hypertable_create - enable_hypertable_compression - enable_cagg_create - enable_policy_create ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.1813839050209377),
 NodeWithScore(node=TextNode(id_='b5583780-3574-11ee-871a-5a8c45d660c8', embedding=None, metadata={'commit': ' 5bba74a2ec083728f8e93e09d03d102568fd72b5', 'author': 'Fabrízio de Royes Mello', 'date': '2023-08-7 19:49:47+-500'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='ec25a09b9dd34ed2650aefc2ce71e1b11fa471ffc43683715de788d202c6cdc8', text='Mon Aug 7 19:49:47 2023 -0300 Fabrízio de Royes Mello Relax strong table lock when refreshing a CAGG When refreshing a Continuous Aggregate we take a table lock on _timescaledb_catalog.continuous_aggs_invalidation_threshold when processing the invalidation logs (the first transaction of the refresh Continuous Aggregate procedure). It means that even two different Continuous Aggregates over two different hypertables will wait each other in the first phase of the refreshing procedure. Also it lead to problems when a pg_dump is running because it take an AccessShareLock on tables so Continuous Aggregate refresh execution will wait until the pg_dump finish.  Improved it by relaxing the strong table-level lock to a row-level lock so now the Continuous Aggregate refresh procedure can be executed in multiple sessions with less locks.  Fix #3554 ', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.23511557892997959)]

接下来我们将介绍如何使用 TimescaleVector 作为[查询引擎](https://gpt-index.readthedocs.io/en/stable/api_reference/query/query_engines.html)。

同样，我们在创建查询引擎时，通过将时间过滤器参数作为 vector_strored_kwargs 传递，使用 TimescaleVector 的时间过滤器将搜索限制在相关时间范围内。

In [ ]

index = VectorStoreIndex.from_vector_store(ts_vector_store)
query_engine = index.as_query_engine(
    vector_store_kwargs=({"start_date": start_dt, "end_date": end_dt})
)

# query_str = "What's new with TimescaleDB? List 3 new features"
query_str = (
    "What's new with TimescaleDB functions? When were these changes made and"
    " by whom?"
)
response = query_engine.query(query_str)
print(str(response))

index = VectorStoreIndex.from_vector_store(ts_vector_store) query_engine = index.as_query_engine( vector_store_kwargs=({"start_date": start_dt, "end_date": end_dt}) ) # query_str = "What's new with TimescaleDB? List 3 new features" query_str = ( "What's new with TimescaleDB functions? When were these changes made and" " by whom?" ) response = query_engine.query(query_str) print(str(response))

TimescaleDB functions have undergone changes recently. These changes were made by Sven Klemm on August 29, 2023. The changes involve adding a compatibility layer for _timescaledb_internal functions. This layer ensures that external callers of these internal functions will not break and allows for more flexibility when migrating.