RAG & Retrieval

Learn Retrieval Augmented Generation (RAG) - the technique that grounds AI responses in your actual data. Covers vector databases, embeddings, chunking, and hybrid search.

What is RAG?

Retrieval Augmented Generation (RAG) is a technique that enhances AI model responses by retrieving relevant information from an external knowledge base and including it in the prompt context. Instead of relying solely on the model's training data, RAG gives the model access to specific, up-to-date, and private information.

RAG Architecture

A RAG system has two phases: an indexing phase (offline, done once) and a query phase (real-time, per request).

1. Embed (Indexing Phase)

   Convert your documents into numerical vectors (embeddings) that capture semantic meaning. Each chunk of text becomes a point in high-dimensional space.

2. Index (Indexing Phase)

   Store these embeddings in a vector database, organized for fast similarity search.

3. Retrieve (Query Phase)

   When a user asks a question, convert the query to an embedding and find the most similar document chunks.

4. Generate (Query Phase)

   Include the retrieved chunks in the prompt context and let the LLM generate a response grounded in your data.

from openai import OpenAI
import chromadb

# 1. Set up vector database
chroma = chromadb.Client()
collection = chroma.get_or_create_collection(
    name="knowledge_base"
)

# 2. Index documents (done once)
documents = [
    "Our return policy allows returns within 30 days...",
    "Shipping takes 3-5 business days for standard...",
    "Premium members get free expedited shipping...",
]
collection.add(
    documents=documents,
    ids=[f"doc_{i}" for i in range(len(documents))]
)

# 3. Query: retrieve relevant docs
query = "How long do I have to return an item?"
results = collection.query(
    query_texts=[query],
    n_results=3
)

# 4. Generate: include context in prompt
context = "\n".join(results["documents"][0])
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": f"""Answer based on this context:

{context}

Question: {query}"""
    }]
)

Vector Databases

Vector databases are specialized databases designed to store and search high-dimensional vectors efficiently. They are the backbone of RAG systems.

Embedding Models

Embedding models convert text into numerical vectors. The quality of your embeddings directly affects retrieval accuracy.

# Using OpenAI embeddings
from openai import OpenAI

client = OpenAI()
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Your text to embed"
)
embedding = response.data[0].embedding
# Returns a list of 1536 floats

# Using Anthropic's Voyage embeddings
import voyageai

client = voyageai.Client()
result = client.embed(
    ["Your text to embed"],
    model="voyage-3"
)
embedding = result.embeddings[0]

Chunking Strategies

Before embedding documents, you need to split them into chunks. The chunking strategy significantly affects retrieval quality:

Retrieval Quality

The quality of your RAG system depends on two metrics:

• Precision: What percentage of retrieved documents are actually relevant? High precision means less noise in context.
• Recall: What percentage of relevant documents were retrieved? High recall means the model has all the information it needs.

Hybrid Search

Hybrid search combines semantic search (embeddings) with keyword search (BM25) for better retrieval. Semantic search understands meaning; keyword search handles exact matches, names, and codes.

def hybrid_search(query, collection, alpha=0.7):
    """
    Combine semantic and keyword search.
    alpha: weight for semantic (1.0 = pure semantic)
    """
    # Semantic search
    semantic_results = collection.semantic_search(
        query, limit=10
    )

    # Keyword search (BM25)
    keyword_results = collection.keyword_search(
        query, limit=10
    )

    # Combine scores with weighted fusion
    combined = reciprocal_rank_fusion(
        semantic_results, keyword_results,
        weights=[alpha, 1 - alpha]
    )

    return combined[:5]  # Top 5 results