How one can Construct a Resilient Utility Utilizing LlamaIndex?

Introduction

LlamaIndex is a well-liked framework for constructing LLM functions. To construct a sturdy software, we have to know how you can depend the embedding tokens earlier than making them, guarantee there are not any duplicates within the vector retailer, get supply knowledge for the generated response, and plenty of different issues. This text will evaluation the steps to construct a resilient software utilizing LlamaIndex.

Studying Goals

• Perceive the important elements and features of the LlamaIndex framework for constructing sturdy LLM functions.
• Learn to create and run an environment friendly ingestion pipeline to rework, parse, and retailer paperwork.
• Achieve information on initializing, saving, and loading paperwork and vector shops to handle persistent knowledge storage successfully.
• Grasp constructing indices and utilizing customized prompts to facilitate environment friendly querying and steady interactions with chat engines.

Conditions

Listed here are just a few stipulations to construct an software utilizing LlamaIndex.

Use the .env file to retailer the OpenAI Key and cargo it from the file

import os
from dotenv import load_dotenv

load_dotenv('/.env') # present path of the .env file
OPENAI_API_KEY = os.environ['OPENAI_API_KEY']

We’ll use Paul Graham’s essay for example doc. It may be downloaded from right here https://github.com/run-llama/llama_index/blob/important/docs/docs/examples/knowledge/paul_graham/paul_graham_essay.txt

How one can Construct an Utility Utilizing LlamaIndex

Load the Knowledge

Step one in constructing an software utilizing LlamaIndex is to load the info.

from llama_index.core import SimpleDirectoryReader
paperwork = SimpleDirectoryReader(input_files=["./data/paul_graham_essay.txt"], 
filename_as_id=True).load_data(show_progress=True)

# 'paperwork' is a listing, which comprises the recordsdata we've got loaded

Allow us to take a look at the keys of the doc object

paperwork[0].to_dict().keys()

# output
"""
dict_keys(['id_', 'embedding', 'metadata', 'excluded_embed_metadata_keys', 
'excluded_llm_metadata_keys', 'relationships', 'text', 'start_char_idx', 
'end_char_idx', 'text_template', 'metadata_template', 'metadata_seperator', 
'class_name'])
"""

We are able to modify the values of these keys as we do for a dictionary. Allow us to take a look at an instance with metadata.

If we need to add extra details about the doc, we will add it to the doc metadata as follows. These metadata tags can be utilized to filter the paperwork.

paperwork[0].metadata.replace({'writer': 'paul_graham'})

paperwork[0].metadata

# output
"""
{'file_path': 'knowledge/paul_graham_essay.txt',
 'file_name': 'paul_graham_essay.txt',
 'file_type': 'textual content/plain',
 'file_size': 75042,
 'creation_date': '2024-04-16',
 'last_modified_date': '2024-04-15',
 'writer': 'paul_graham'}
"""

Ingestion Pipeline

With the ingestion pipeline, we will carry out all the info transformations, comparable to parsing the doc into nodes, extracting metadata for the nodes, creating embeddings, storing the info within the doc retailer, and storing the embeddings and textual content of the nodes within the vector retailer. This enables us to maintain every thing wanted to make the info obtainable for indexing in a single place.

Extra importantly, utilizing the doc retailer and vector retailer will be sure that duplicate embeddings are usually not created if we save and cargo the doc retailer and vector shops and run the ingestion pipeline on the identical paperwork.

Token Counting

The subsequent step in constructing an software utilizing LlamaIndex is token counting.

import the dependencies
import nest_asyncio

nest_asyncio.apply()

import tiktoken

from llama_index.core.callbacks import CallbackManager, TokenCountingHandler

from llama_index.core import MockEmbedding
from llama_index.core.llms import MockLLM

from llama_index.core.node_parser import SentenceSplitter,HierarchicalNodeParser

from llama_index.core.ingestion import IngestionPipeline

from llama_index.core.extractors import TitleExtractor, SummaryExtractor

Initialize the token counter

token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode,
    verbose=True
)

Now, we will transfer on to construct an ingestion pipeline utilizing MockEmbedding and MockLLM.

mock_pipeline = IngestionPipeline(
 transformations = [SentenceSplitter(chunk_size=512, chunk_overlap=64),
 TitleExtractor(llm=MockLLM(callback_manager=CallbackManager([token_counter]))),
 MockEmbedding(embed_dim=1536, callback_manager=CallbackManager([token_counter]))])
 
nodes = mock_pipeline.run(paperwork=paperwork, show_progress=True, num_workers=-1)

The above code applies a sentence splitter to the paperwork to create nodes, then makes use of mock embedding and llm fashions for metadata extraction and embedding creation.

Then, we will examine the token counts

# this returns the depend of embedding tokens 
token_counter.total_embedding_token_count

# this returns the depend of llm tokens 
token_counter.total_llm_token_count

# token counter is cumulative. Once we need to set the token counts to zero, we will use this
token_counter.reset_counts()

We are able to strive totally different node parsers and metadata extractors to find out what number of tokens it’s going to take.

Create Doc and Vector Shops

The subsequent step in constructing an software utilizing LlamaIndex is to create doc and vector shops.

from llama_index.embeddings.openai import OpenAIEmbedding

from llama_index.core.storage.docstore import SimpleDocumentStore

from llama_index.vector_stores.chroma import ChromaVectorStore

import chromadb

Now we will initialize the doc and vector shops

doc_store = SimpleDocumentStore()

# point out the trail, the place vector retailer is saved
chroma_client = chromadb.PersistentClient(path="./chroma_db")

# we are going to create a group if would not already exists
chroma_collection = chroma_client.get_or_create_collection("paul_essay")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

pipeline = IngestionPipeline(
    transformations = [SentenceSplitter(chunk_size=512, chunk_overlap=128),
    OpenAIEmbedding(model_name="text-embedding-3-small", 
              callback_manager=CallbackManager([token_counter]))],
    docstore=doc_store,
    vector_store=vector_store
)
nodes = pipeline.run(paperwork=paperwork, show_progress=True, num_workers=-1)

As soon as we run the pipeline, embeddings are saved within the vector retailer for the nodes. We additionally want to save lots of the doc retailer.

doc_store.persist('./doc storage/doc_store.json')

# we will additionally examine the embedding token depend
token_counter.total_embedding_token_count

Now, we will restart the kernel to load the saved shops.

Load the Doc and Vector Shops

Now, allow us to import the mandatory strategies, as talked about above.

# load the doc retailer
doc_store = SimpleDocumentStore.from_persist_path('./doc storage/doc_store.json')

# load the vector retailer
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("paul_essay")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

Now, you initialize the above pipeline once more and run it. Nevertheless, it doesn’t create embeddings as a result of the system has already processed and saved the doc. So, we add any new doc to a folder, load all of the paperwork, and run the pipeline, creating embeddings just for the brand new doc.

We are able to examine it with the next

# hash of the doc
paperwork[0].hash

# you will get the doc identify from the doc_store
for i in doc_store.docs.keys():
    print(i)
    
# hash of the doc within the doc retailer
doc_store.docs['data/paul_graham_essay.txt'].hash

# When each of these hashes match, duplicate embeddings are usually not created. 

Look into the Vector Retailer

Let’s see what’s saved within the vector retailer.

chroma_collection.get().keys()
# output
# dict_keys(['ids', 'embeddings', 'metadatas', 'documents', 'uris', 'data'])

chroma_collection.get()['metadatas'][0].keys()
# output
# dict_keys(['_node_content', '_node_type', 'creation_date', 'doc_id', 
  'document_id', 'file_name', 'file_path', 'file_size', 
  'file_type', 'last_modified_date', 'ref_doc_id'])

# this can return ids, metadatas, and paperwork of the nodes within the assortment
chroma_collection.get() 

How do we all know which node corresponds to which doc? We are able to look into the metadata node_content

ids = chroma_collection.get()['ids']

# this can print doc identify for every node
for i in ids:
    knowledge = json.hundreds(chroma_collection.get(i)['metadatas'][0]['_node_content'])
    print(knowledge['relationships']['1']['node_id'])

# this can embrace the embeddings of the node together with metadata and textual content
chroma_collection.get(ids=ids[0],embrace=['embeddings', 'metadatas', 'documents'])

# we will additionally filter the gathering
chroma_collection.get(ids=ids, the place={'file_size': {'$gt': 75040}}, 
   where_document={'$comprises': 'paul'}, embrace=['metadatas', 'documents'])

Querying

from llama_index.llms.openai import OpenAI

from llama_index.core.retrievers import VectorIndexRetriever

from llama_index.core import get_response_synthesizer

from llama_index.core.response_synthesizers.kind import ResponseMode

from llama_index.core.query_engine import RetrieverQueryEngine

from llama_index.core.chat_engine import (ContextChatEngine, 
CondenseQuestionChatEngine, CondensePlusContextChatEngine)

from llama_index.core.storage.chat_store import SimpleChatStore

from llama_index.core.reminiscence import ChatMemoryBuffer

from llama_index.core import PromptTemplate

from llama_index.core.chat_engine.sorts import ChatMode

from llama_index.core.llms import ChatMessage, MessageRole
from llama_index.core import ChatPromptTemplate

Now, we will construct an index from the vector retailer. An index is a knowledge construction that facilitates the fast retrieval of related context for a consumer question.

# outline the index
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

# outline a retriever
retriever = VectorIndexRetriever(index=index, similarity_top_k=3)

Within the above code, the retriever retrieves the highest 3 related nodes to the question we give.

If we would like the LLM to reply the question based mostly on solely the context offered and never anything, we will use the customized prompts accordingly.

qa_prompt_str = (
    "Context data is beneath.n"
    "---------------------n"
    "{context_str}n"
    "---------------------n"
    "Given the context data and never prior information, "
    "reply the query: {query_str}n"
)
chat_text_qa_msgs = [
    ChatMessage(role=MessageRole.SYSTEM,
 content=("Only answer the question, if the question is answerable with the given context. 
        Otherwise say that question can't be answered using the context"),
                ),
    ChatMessage(role=MessageRole.USER, content=qa_prompt_str)]
    
text_qa_template = ChatPromptTemplate(chat_text_qa_msgs)

Now, we will outline the response synthesizer, which passes the context and queries to the LLM to get the response. We are able to additionally add a token counter as a callback supervisor to maintain observe of the tokens used.

gpt_3_5 = OpenAI(mannequin="gpt-3.5-turbo")

response_synthesizer = get_response_synthesizer(llm = gpt_3_5, response_mode=ResponseMode.COMPACT, 
                                                text_qa_template=text_qa_template, 
                                                callback_manager=CallbackManager([token_counter]))

Now, we will mix the retriever and response_synthesizer as a question engine that takes the question.

query_engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=response_synthesizer)
    
# ask a question
Response = query_engine.question("who's paul graham?")

# response textual content
Response.response

To know which textual content is used to generate this response, we will use the next code

for i, node in enumerate(Response.source_nodes):
    print(f"textual content of the node {i}")
    print(node.textual content)
    print("------------------------------------n")

Equally, we will strive totally different question engines.

Chatting

If we need to converse with our knowledge, we have to retailer the earlier queries and the responses reasonably than asking remoted queries.

chat_store = SimpleChatStore()

chat_memory = ChatMemoryBuffer.from_defaults(token_limit=5000, chat_store=chat_store, llm=gpt_3_5)

system_prompt = "Reply the query solely based mostly on the context offered"

chat_engine = CondensePlusContextChatEngine(retriever=retriever, 
              llm=gpt_3_5, system_prompt=system_prompt, reminiscence=chat_memory)

Within the above code, we’ve got initialized chat_store and created the chat_memory object with a token restrict of 5000. We are able to additionally present a system_prompt and different prompts.

Then, we will create a chat engine by additionally together with retriever and chat_memory

We are able to get the response as follows

streaming_response = chat_engine.stream_chat("Who's Paul Graham?")

for token in streaming_response.response_gen:
    print(token, finish="")

We are able to learn the chat historical past with given code

for i in chat_memory.chat_store.retailer['chat_history']:
    print(i.position.identify)
    print(i.content material)

Now we will save and restore the chat_store as wanted

chat_store.persist(persist_path="chat_store.json")
chat_store = SimpleChatStore.from_persist_path(
    persist_path="chat_store.json"
)

This manner, we will construct sturdy RAG functions utilizing the LlamaIndex framework and check out varied superior retrievers and re-rankers.

Additionally Learn: Construct a RAG Pipeline With the LLama Index

Conclusion

The LlamaIndex framework presents a complete answer for constructing resilient LLM functions, guaranteeing environment friendly knowledge dealing with, persistent storage, and enhanced querying capabilities. It’s a worthwhile instrument for builders working with massive language fashions. The important thing takeaways from this information on LlamaIndex are:

• The LlamaIndex framework permits sturdy knowledge ingestion pipelines, guaranteeing organized doc parsing, metadata extraction, and embedding creation whereas stopping duplicates.
• By successfully managing doc and vector shops, LlamaIndex ensures knowledge consistency and facilitates simple retrieval and storage of doc embeddings and metadata.
• The framework helps constructing indices and customized question engines, enabling fast context retrieval for consumer queries and steady interactions by chat engines.

Steadily Requested Questions