Hybrid RAG

What is Hybrid RAG?

Hybrid RAG combines two complementary retrieval methods: sparse retrieval (keyword-based, like BM25) and dense retrieval (vector-based, using embeddings). Each method has distinct strengths, and combining them produces more robust retrieval. Example Scenario: A developer searches for the exact error code "ERR_CONNECTION_RESET" and also wants semantically related troubleshooting information. Pure vector search might find semantically related documents about connection errors but miss the exact error code match. Pure keyword search would find the exact match but miss related troubleshooting guides that use different terminology. Hybrid search finds both.

Sparse vs Dense Retrieval

Sparse Retrieval (BM25): • Based on exact term matching with TF-IDF-like scoring • Excels at matching specific identifiers, names, error codes, and technical terms • Fast and well-understood; no neural network required • Struggles with synonyms and paraphrasing ("car" won't match "automobile") • Represented as sparse vectors (mostly zeros, with non-zero entries for matching terms) Dense Retrieval (Vector Search): • Based on semantic similarity using learned embeddings • Excels at understanding meaning, synonyms, and paraphrased queries • Can match "climate change effects" with documents about "global warming impact" • May miss exact identifiers or rare technical terms • Represented as dense vectors (every dimension has a non-zero value) Why Combine Them? Neither method alone is sufficient for all query types. Hybrid search leverages the strengths of both: • BM25 handles exact matches, acronyms, and specific identifiers • Vector search handles semantic similarity and natural language variation • Together, they provide more robust retrieval across diverse query types

When to Use Hybrid Search

Hybrid search is particularly valuable when your corpus contains both natural language text and technical identifiers (error codes, product IDs, API names), or when users mix natural language with exact terms in their queries.

How BM25 Works

BM25 (Best Matching 25) is a probabilistic ranking function that scores documents based on: Term Frequency (TF): How often the query term appears in the document. More occurrences generally mean higher relevance, but with diminishing returns (the 10th occurrence matters less than the 1st). Inverse Document Frequency (IDF): How rare the term is across the entire corpus. Rare terms are more informative than common terms. The word "the" appears everywhere and is nearly useless for retrieval, while "ERR_CONNECTION_RESET" is highly specific. Document Length Normalization: Longer documents naturally contain more term occurrences. BM25 adjusts for this so that a short document containing a term is scored higher than a long document where the same term appears with the same frequency. The BM25 formula combines these factors with tunable parameters k1 (term frequency saturation) and b (length normalization).

bm25_retrieval.pypython

from langchain_community.retrievers import BM25Retriever

# Create BM25 retriever from documents
bm25_retriever = BM25Retriever.from_documents(
    documents=chunks,
    k=4  # Return top 4 results
)

# BM25 excels at exact term matching
results = bm25_retriever.invoke("ERR_CONNECTION_RESET")
for doc in results:
    print(f"Score match: {doc.page_content[:100]}...")

Reciprocal Rank Fusion (RRF)

When you have results from multiple retrievers, you need a strategy to merge them into a single ranked list. Reciprocal Rank Fusion (RRF) is the most popular method. How RRF Works: For each document, RRF calculates a score based on its rank position in each retriever's results: score(doc) = Σ 1 / (k + rank_i(doc)) Where k is a constant (typically 60) and rank_i is the document's rank in retriever i. Documents ranked highly by multiple retrievers get higher combined scores. Why RRF? • It doesn't require normalized scores from different retrievers (which use different scoring scales) • It's simple to implement and robust • Documents that appear in multiple result sets are naturally boosted • The constant k prevents the top-ranked result from dominating

Tip

Start with equal weights (0.5/0.5) and adjust based on your query patterns. If most queries are natural language, weight dense retrieval higher. If queries often contain exact identifiers, weight BM25 higher.

hybrid_rag.pypython

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# Dense retriever (vector search)
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
dense_retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# Sparse retriever (BM25)
sparse_retriever = BM25Retriever.from_documents(chunks, k=4)

# Combine with Ensemble (uses RRF by default)
hybrid_retriever = EnsembleRetriever(
    retrievers=[dense_retriever, sparse_retriever],
    weights=[0.5, 0.5]  # Equal weighting
)

# This retrieves from both and merges results
results = hybrid_retriever.invoke("ERR_CONNECTION_RESET troubleshooting steps")
for doc in results:
    print(f"Hybrid result: {doc.page_content[:100]}...")

Optional: Reranking

After combining results from multiple retrievers, a reranking step can further improve result quality. A cross-encoder reranker takes each query-document pair and produces a precise relevance score. Why Rerank? • First-stage retrievers (both BM25 and embedding search) are optimized for speed over precision • Cross-encoders are much more accurate because they process the query and document jointly • Reranking a small set of candidates (e.g., top 20) is computationally feasible Popular Rerankers: • Cohere Rerank API • Cross-encoder models from Sentence Transformers (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) • Jina Reranker

reranking.pypython

from sentence_transformers import CrossEncoder

# Load a cross-encoder reranker
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

# Get initial results from hybrid retriever
initial_results = hybrid_retriever.invoke(query)

# Score each document against the query
pairs = [(query, doc.page_content) for doc in initial_results]
scores = reranker.predict(pairs)

# Sort by reranker score
reranked = sorted(zip(scores, initial_results), key=lambda x: x[0], reverse=True)

# Use top results for generation
top_docs = [doc for _, doc in reranked[:4]]

Learning Objectives