LlamaIndex

Overview

LlamaIndex is a data framework created by Jerry Liu, focused on connecting LLMs with private data. Its core is data ingestion, indexing, and retrieval, enabling LLMs to access and leverage large amounts of documents. LlamaIndex provides rich index types (list, tree, keyword, vector, etc.) and query engines, supporting complex retrieval strategies. Unlike LangChain's general-purpose framework, LlamaIndex is more focused on RAG scenarios, providing deeper data integration capabilities.

Installation

LlamaIndex is installed via pip, with the core package providing basic functionality. Additional integration packages are installed as needed. Python 3.9+ is recommended.

# Install core package
pip install llama-index

# Install specific integrations
pip install llama-index-llms-openai
pip install llama-index-embeddings-openai
pip install llama-index-vector-stores-chroma

# Or install the full package
pip install llama-index[full]

Documents

Document is the basic data unit of LlamaIndex. It supports multiple data sources: PDF, web pages, databases, APIs, and more. Documents are split into Nodes for indexing and retrieval. LlamaIndex provides 100+ data loaders.

from llama_index.core import Document, SimpleDirectoryReader

# Load from files
documents = SimpleDirectoryReader("./data").load_data()

# Create manually
doc = Document(text="This is document content", metadata={"source": "manual"})

# Load from web pages
from llama_index.readers.web import SimpleWebPageReader
docs = SimpleWebPageReader().load_data(["https://example.com"])

Indices

Index is the core of LlamaIndex, organizing documents for efficient retrieval. It supports multiple index types: VectorStoreIndex, SummaryIndex, KeywordTableIndex, KnowledgeGraphIndex, and more. Each index type is suitable for different query scenarios.

from llama_index.core import VectorStoreIndex, SummaryIndex

# Vector index (most commonly used)
vector_index = VectorStoreIndex.from_documents(documents)

# Summary index
summary_index = SummaryIndex.from_documents(documents)

# Persist
vector_index.storage_context.persist(persist_dir="./storage")

Query Engines

Query Engine is the query interface that converts user queries into answers. It supports multiple query modes: retrieval, summary, routing, sub-queries, and more. Multiple query engines can be combined to implement complex logic.

from llama_index.core.query_engine import RetrieverQueryEngine

# Basic query engine
query_engine = vector_index.as_query_engine()
response = query_engine.query("What is RAG?")

# Streaming response
streaming_engine = vector_index.as_query_engine(streaming=True)
response = streaming_engine.query("question")
for text in response.response_gen:
    print(text, end="")

RAG Pipelines

LlamaIndex provides complete RAG pipeline building capabilities. It includes data ingestion, chunking, embedding, indexing, retrieval, reranking, generation, and other steps. It supports advanced RAG techniques such as sentence window retrieval, auto-merging retrieval, hybrid retrieval, and more.

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import VectorStoreIndex

# Advanced RAG configuration
splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=20)
nodes = splitter.get_nodes_from_documents(documents)

index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine(
    similarity_top_k=5,
    response_mode="compact"
)

Agents

LlamaIndex provides Agent functionality, supporting autonomous decision-making and tool invocation. Agents can use query engines as tools to implement complex reasoning and retrieval. Types such as OpenAI Agent and ReAct Agent are supported.

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool

# Use query engine as a tool
tool = QueryEngineTool.from_defaults(
    query_engine=query_engine,
    name="knowledge_base",
    description="Query the knowledge base for information"
)

agent = ReActAgent.from_tools([tool], llm=llm)
response = agent.chat("Answer the question based on the documents")

Tools

LlamaIndex provides a rich set of tools, including query engine tools, function tools, Spec tools, and more. Tools can be combined to build powerful agents. Custom tool development is supported.

from llama_index.core.tools import FunctionTool

# Custom function tool
def search_database(query: str) -> str:
    """Search the database."""
    # Implement search logic
    return results

tool = FunctionTool.from_defaults(fn=search_database)

# Use the tool
agent = ReActAgent.from_tools([tool], llm=llm)

Evaluation

LlamaIndex provides an evaluation framework to assess the quality of RAG systems. It supports evaluating retrieval relevance, answer accuracy, faithfulness, and more. Evaluation helps optimize RAG pipeline parameters and configuration.

from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator

faithfulness = FaithfulnessEvaluator(llm=llm)
relevancy = RelevancyEvaluator(llm=llm)

# Evaluate the answer
faith_result = faithfulness.evaluate_response(response=response)
rel_result = relevancy.evaluate_response(query="question", response=response)

Deployment

LlamaIndex applications can be deployed as web services, APIs, CLI tools, and more. Integration with FastAPI and Flask is supported. LlamaCloud provides managed services, simplifying deployment and scaling. Deployment to cloud platforms is also supported.

# FastAPI deployment
from fastapi import FastAPI
from llama_index.core import VectorStoreIndex

app = FastAPI()
index = VectorStoreIndex.load_from_persist_dir("./storage")
engine = index.as_query_engine()

@app.post("/query")
async def query(question: str):
    response = engine.query(question)
    return {"answer": str(response)}

Persist the index to disk so the API can reload it without rebuilding on every start.

Configuration

LlamaIndex is configured in Python with integration packages. Set LLM API keys as environment variables and pick an embedding model + vector store. Index type (VectorStoreIndex, SummaryIndex, etc.) and query engine settings (similarity_top_k, response_mode) are the main tuning knobs. Storage persists to a directory via storage_context.

# .env
OPENAI_API_KEY=your-key

# Python: model + embedding + vector store
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import VectorStoreIndex, Settings

Settings.llm = OpenAI(model="gpt-4o")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# Index with tuning
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(
    similarity_top_k=5,
    response_mode="compact"
)

# Persist
index.storage_context.persist(persist_dir="./storage")

Tune similarity_top_k and chunk_size together—larger chunks need fewer retrieved nodes.

FAQ

Common questions cover LlamaIndex vs LangChain, model choice, cost, RAG tuning, and index types. LlamaIndex is RAG-focused with deeper data ingestion tooling; LangChain is a more general framework. VectorStoreIndex is the default for most RAG apps.

Q: LlamaIndex vs LangChain?
A: LlamaIndex is RAG-focused with richer indexing/retrieval abstractions;
   LangChain is a general LLM framework. They can be used together.

Q: Which index type should I use?
A: VectorStoreIndex for most RAG; SummaryIndex for whole-doc summaries;
   KnowledgeGraphIndex for relationship-heavy data.

Q: How do I cut costs?
A: Use a smaller embedding model, lower similarity_top_k, and cache
   embeddings in a persisted vector store.

Q: How do I improve RAG quality?
A: Tune chunk_size/chunk_overlap, use sentence-window or auto-merging
   retrieval, and add a reranker.

Q: Which models are supported?
A: Any provider with an integration package: OpenAI, Anthropic, Ollama,
   Google, and more.

For better RAG, add a reranker (e.g. Cohere Rerank) on top of vector retrieval.