Conversational RAG

What is Conversational RAG?

Conversational RAG extends basic RAG to handle multi-turn conversations. In a basic RAG system, each question is processed independently — the system has no memory of previous interactions. This fails when users ask follow-up questions that reference earlier messages. Example: • User: "What is Python?" • Assistant: "Python is a high-level, interpreted programming language known for..." • User: "What are its advantages?" In basic RAG, the second question "What are its advantages?" would be sent directly to the retriever. Since the retriever doesn't know what "its" refers to, it would retrieve irrelevant documents. Conversational RAG solves this by rewriting the question to include the necessary context: "What are the advantages of Python?"

Question Rewriting

The core technique in Conversational RAG is question rewriting (also called query reformulation or contextualization). Before sending a query to the retriever, an LLM rewrites the question to be self-contained by incorporating relevant context from the chat history. How It Works: 1. The system receives a new question along with the chat history 2. An LLM analyzes the question and the history to determine if the question contains ambiguous references 3. The LLM rewrites the question to be standalone — replacing pronouns and references with explicit terms 4. The rewritten question is used for retrieval Example Rewriting: • Chat History: [User: "What is Django?", Assistant: "Django is a Python web framework..."] • New Question: "How does it handle databases?" • Rewritten: "How does Django handle databases?" This approach is more efficient than stuffing the entire chat history into the retriever query, which would add noise and reduce retrieval precision.

question_rewriting.pypython

from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4", temperature=0)

contextualize_prompt = ChatPromptTemplate.from_messages([
    ("system", """Given the chat history and the latest user question, 
    reformulate the question to be standalone — it should be understandable 
    without the chat history. Do NOT answer the question, just reformulate it 
    if needed. If it's already standalone, return it unchanged."""),
    ("human", "Chat History:\n{chat_history}\n\nLatest Question: {question}"),
])

# Example usage
chat_history = """User: What is Django?
Assistant: Django is a high-level Python web framework that encourages rapid development."""

question = "How does it handle databases?"

chain = contextualize_prompt | llm
result = chain.invoke({
    "chat_history": chat_history,
    "question": question
})
print(result.content)
# Output: "How does Django handle databases?"

Conversation Memory

Managing conversation history effectively is important for Conversational RAG. You need to decide how much history to retain and how to format it for the LLM. Memory Strategies: Buffer Memory: Store the last N messages verbatim. Simple but can grow large. Window Memory: Keep only the last K conversation turns, discarding older ones. Limits context size but may lose important early context. Summary Memory: Periodically summarize older parts of the conversation, keeping recent messages verbatim. Balances context retention with token efficiency. Token-Limited Memory: Keep as many recent messages as fit within a token budget. Practical Considerations: • The entire conversation context (history + retrieved documents + question) must fit within the LLM's context window • Very long histories can dilute retrieval quality — summarize when possible • Consider which parts of the history are relevant to the current question

Tip

For production systems, persist conversation history in a database rather than in-memory. This allows conversations to resume across sessions and supports analytics.

conversational_rag_chain.pypython

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferWindowMemory
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import Chroma

llm = ChatOpenAI(model="gpt-4", temperature=0)

# Set up memory (keep last 5 exchanges)
memory = ConversationBufferWindowMemory(
    k=5,
    memory_key="chat_history",
    return_messages=True,
    output_key="answer"
)

# Create conversational retrieval chain
conv_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    memory=memory,
    return_source_documents=True,
    verbose=True
)

# Multi-turn conversation
response1 = conv_chain.invoke({"question": "What is Python?"})
print("A1:", response1["answer"])

response2 = conv_chain.invoke({"question": "What are its main advantages?"})
print("A2:", response2["answer"])

response3 = conv_chain.invoke({"question": "How does it compare to Java?"})
print("A3:", response3["answer"])

Conversation Flow Architecture

The complete Conversational RAG flow: 1. User sends a message — could be a new question or a follow-up 2. Check if rewriting is needed — analyze whether the question contains references to earlier messages 3. Rewrite the question — if needed, use the LLM to create a standalone version 4. Retrieve relevant documents — using the rewritten (standalone) question 5. Generate answer — the LLM receives the original question (for natural conversation), the chat history (for conversational context), and the retrieved documents 6. Update memory — store the new exchange in conversation history 7. Return response — deliver the answer with source citations This approach preserves the natural conversational flow (users see their original questions) while using optimized standalone queries for retrieval.

Learning Objectives

What is Conversational RAG?

Question Rewriting

Conversation Memory

Conversation Flow Architecture

Key Takeaways