IntermediateEstimated time: 50 mins

Conversational RAG with Memory

Implement multi-turn conversational question answering with query reformulation and chat history.

Tech Stack:PythonLangChainChromaDBChat History MemoryOpenAI / Anthropic

System Architecture

Query + Chat History ➔ Contextualize Query Prompt ➔ Standalone Reformulated Question ➔ Vector Retriever ➔ QA Generation Chain ➔ Message History Store

Prerequisites

  • Completion of Basic RAG / Chat with PDF
  • Understanding of multi-turn chat history vs stateless API calls

Installation Commands

terminalbash
pip install langchain langchain-community langchain-openai chromadb

Folder Structure

Directory Layouttext
conversational-rag/
├── main.py
├── memory_store.py
└── requirements.txt

Step-by-Step Implementation

1

The Need for Query Reformulation

When users ask follow-up questions like "What are its main benefits?", the pronoun "its" fails in vector search unless rewritten using chat history.

main.py (Reformulation Chain)python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.chains import create_history_aware_retriever

llm = ChatOpenAI(model="gpt-4o", temperature=0)

contextualize_q_system_prompt = (
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question which can be understood "
    "without the chat history. Do NOT answer the question, "
    "just reformulate it if needed and otherwise return it as is."
)

contextualize_q_prompt = ChatPromptTemplate.from_messages([
    ("system", contextualize_q_system_prompt),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}"),
])

# history_aware_retriever produces a standalone query first!
history_retriever = create_history_aware_retriever(llm, retriever, contextualize_q_prompt)
2

Building the Full History-Aware QA Chain

Connect the history-aware retriever with the answer generation chain and wrap it with RunnableWithMessageHistory.

main.py (Full Pipeline)python
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

qa_system_prompt = (
    "You are a helpful knowledge assistant. Answer questions using the retrieved context.\n\n{context}"
)
qa_prompt = ChatPromptTemplate.from_messages([
    ("system", qa_system_prompt),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}"),
])

question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)
rag_chain = create_retrieval_chain(history_retriever, question_answer_chain)

# Session-based memory store
store = {}
def get_session_history(session_id: str):
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

conversational_rag = RunnableWithMessageHistory(
    rag_chain,
    get_session_history,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="answer",
)

Example Execution (Input & Output)

Input Query:
Turn 1: "What is PostgreSQL?" Turn 2: "What indexing methods does it support?"
Generated Grounded Output:
Turn 2 Reformulation: "What indexing methods does PostgreSQL support?" Answer: PostgreSQL supports B-tree, Hash, GiST, SP-GiST, GIN, and BRIN indexes.

Why this works: The system accurately mapped the pronoun "it" back to "PostgreSQL" from Turn 1 before querying the vector store.

Troubleshooting Common Gotchas

Chat history consumes too many tokens over long sessions

Apply token sliding windowing or summarize older turns with ConversationSummaryBufferMemory.

Known Limitations

  • •Two LLM calls per turn (one for question rewriting, one for answer synthesis), increasing latency.

Security Considerations

  • •Isolate session history by authenticated user ID to prevent cross-tenant data leaks.
Back to all projects