Nomic Embedding¶

Nomic 已发布 v1.5 🪆🪆🪆，它具备可变大小的嵌入能力，采用套娃学习（matryoshka learning），上下文长度为 8192，嵌入维度介于 64 和 768 之间。

在本 Notebook 中，我们将探索在不同维度下使用 Nomic v1.5 嵌入向量。

安装¶

In [ ]

%pip install -U llama-index llama-index-embeddings-nomic

%pip install -U llama-index llama-index-embeddings-nomic

设置 API 密钥¶

In [ ]

nomic_api_key = "<NOMIC API KEY>"

nomic_api_key = ""

In [ ]

import nest_asyncio

nest_asyncio.apply()

from llama_index.embeddings.nomic import NomicEmbedding

import nest_asyncio nest_asyncio.apply() from llama_index.embeddings.nomic import NomicEmbedding

使用 128 维¶

In [ ]

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=128, model_name="nomic-embed-text-v1.5", ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]

print(len(embedding))

print(len(embedding))

128

In [ ]

embedding[:5]

embedding[:5]

Out[ ]

[0.05569458, 0.057922363, -0.30126953, -0.09832764, 0.05947876]

使用 256 维¶

In [ ]

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=256,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=256, model_name="nomic-embed-text-v1.5", ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]

print(len(embedding))

print(len(embedding))

256

In [ ]

embedding[:5]

embedding[:5]

Out[ ]

[0.044708252, 0.04650879, -0.24182129, -0.07897949, 0.04776001]

使用 768 维¶

In [ ]

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=768,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=768, model_name="nomic-embed-text-v1.5", ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]

print(len(embedding))

print(len(embedding))

768

In [ ]

embedding[:5]

embedding[:5]

Out[ ]

[0.027282715, 0.028381348, -0.14758301, -0.048187256, 0.029144287]

您仍然可以使用 v1 Nomic 嵌入向量¶

它具有 768 个固定嵌入维度

In [ ]

embed_model = NomicEmbedding(
    api_key=nomic_api_key, model_name="nomic-embed-text-v1"
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

embed_model = NomicEmbedding( api_key=nomic_api_key, model_name="nomic-embed-text-v1" ) embedding = embed_model.get_text_embedding("Nomic Embeddings")

In [ ]

print(len(embedding))

print(len(embedding))

768

In [ ]

embedding[:5]

embedding[:5]

Out[ ]

[0.0059013367, 0.03744507, 0.0035305023, -0.047180176, 0.0154418945]

让我们使用 Nomic v1.5 嵌入向量构建端到端 RAG 管道。¶

我们将在生成步骤中使用 OpenAI。

设置嵌入模型和 LLM。¶

In [ ]

from llama_index.core import settings
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

import os

os.environ["OPENAI_API_KEY"] = "<YOUR OPENAI API KEY>"

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)

llm = OpenAI(model="gpt-3.5-turbo")

settings.llm = llm
settings.embed_model = embed_model

from llama_index.core import settings from llama_index.core import VectorStoreIndex, SimpleDirectoryReader from llama_index.llms.openai import OpenAI import os os.environ["OPENAI_API_KEY"] = "" embed_model = NomicEmbedding( api_key=nomic_api_key, dimensionality=128, model_name="nomic-embed-text-v1.5", ) llm = OpenAI(model="gpt-3.5-turbo") settings.llm = llm settings.embed_model = embed_model

下载数据¶

In [ ]

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

--2024-02-16 18:37:03--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: 'data/paul_graham/paul_graham_essay.txt'

data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.02s   

2024-02-16 18:37:03 (3.87 MB/s) - 'data/paul_graham/paul_graham_essay.txt' saved [75042/75042]

加载数据¶

In [ ]

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

创建索引¶

In [ ]

index = VectorStoreIndex.from_documents(documents)

index = VectorStoreIndex.from_documents(documents)

查询引擎¶

In [ ]

query_engine = index.as_query_engine()

query_engine = index.as_query_engine()

In [ ]

response = query_engine.query("what did author do growing up?")
print(response)

response = query_engine.query("what did author do growing up?") print(response)

The author, growing up, worked on writing and programming. They wrote short stories and also tried writing programs on an IBM 1401 computer. Later, they got a microcomputer and started programming more extensively, writing simple games and a word processor.