Agentic RAG

What is Agentic RAG?

Agentic RAG replaces the fixed pipeline of Basic RAG with an intelligent agent that can make dynamic decisions about when and how to retrieve information. Instead of always following the same retrieve-then-generate pattern, an agent can: • Decide whether retrieval is needed at all • Choose which tools or data sources to query • Evaluate the quality of retrieved results • Rewrite queries when initial retrieval fails • Perform multiple rounds of retrieval • Combine information from different sources • Decide when it has enough information to generate an answer Example Scenario: An AI research assistant receives a complex question. It searches the document collection, evaluates whether the results are sufficient, rewrites the query to be more specific when needed, performs another retrieval, and only generates an answer when it has gathered enough relevant evidence.

Agent Behavior is Defined by the Workflow

The agent's capabilities depend entirely on the tools, prompts, and workflow logic you implement. An "agentic" system is only as smart as its design — it does not automatically know how to handle every situation.

Agent Architecture

An agentic RAG system typically consists of: Agent Core: The LLM that acts as the decision-maker, interpreting queries and choosing actions based on the current state. Tools: Functions the agent can call, such as: • Document retriever (search the vector store) • Web search (for external information) • Calculator or code executor (for computational questions) • Database query tool (for structured data) State Management: Tracks the current conversation state, retrieved documents, attempted queries, and evaluation results. Routing Logic: Conditional logic that determines the next step based on the current state. For example: • If retrieved documents are relevant → generate answer • If retrieved documents are irrelevant → rewrite query and retry • If the question is simple → answer directly without retrieval • If the question requires multiple sources → perform multiple retrievals Stopping Conditions: Criteria for when the agent should stop iterating: • Sufficient relevant documents found • Maximum number of retries reached • Confidence threshold met

Implementation with LangGraph

LangGraph is a library for building stateful, graph-based agent workflows. It represents the workflow as a directed graph where nodes are processing steps and edges define the flow (including conditional edges for branching). Why LangGraph? • Explicit control flow (no magic — you define every step) • State management built-in • Conditional routing between steps • Support for loops and iterative refinement • Debugging and visualization tools The following example builds an agentic RAG workflow that: 1. Receives a question 2. Retrieves documents 3. Evaluates document relevance 4. Either generates an answer (if relevant) or rewrites the query and retries

agentic_rag.pypython

from langgraph.graph import StateGraph, END
from typing import TypedDict, List, Annotated
from langchain_core.documents import Document

class AgentState(TypedDict):
    question: str
    documents: List[Document]
    generation: str
    retry_count: int
    rewritten_question: str

def retrieve(state: AgentState) -> dict:
    """Retrieve documents for the current question."""
    query = state.get("rewritten_question") or state["question"]
    docs = retriever.invoke(query)
    return {"documents": docs}

def grade_documents(state: AgentState) -> str:
    """Evaluate if retrieved documents are relevant."""
    question = state["question"]
    docs = state["documents"]
    
    # Use LLM to evaluate relevance
    relevant = evaluate_relevance(question, docs)
    
    if relevant:
        return "generate"
    elif state.get("retry_count", 0) >= 2:
        return "generate"  # Give up retrying, use what we have
    else:
        return "rewrite"

def rewrite_query(state: AgentState) -> dict:
    """Rewrite the question for better retrieval."""
    question = state["question"]
    retry = state.get("retry_count", 0) + 1
    
    rewritten = llm.invoke(
        f"Rewrite this question to be more specific for document search: {question}"
    ).content
    
    return {"rewritten_question": rewritten, "retry_count": retry}

def generate(state: AgentState) -> dict:
    """Generate answer from retrieved documents."""
    docs_text = "\n\n".join(d.page_content for d in state["documents"])
    answer = llm.invoke(
        f"Context: {docs_text}\n\nQuestion: {state['question']}\n\nAnswer:"
    ).content
    return {"generation": answer}

# Build the graph
workflow = StateGraph(AgentState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("rewrite", rewrite_query)
workflow.add_node("generate", generate)

workflow.set_entry_point("retrieve")
workflow.add_conditional_edges(
    "retrieve",
    grade_documents,
    {"generate": "generate", "rewrite": "rewrite"}
)
workflow.add_edge("rewrite", "retrieve")  # Retry after rewriting
workflow.add_edge("generate", END)

app = workflow.compile()
result = app.invoke({"question": "How does Django connect to databases?"})

Routing Strategies

Agents can use different strategies to route queries: LLM-Based Routing: Use the LLM to classify the question and determine the best action. This is flexible but adds latency (an extra LLM call). Rule-Based Routing: Use heuristic rules (keyword matching, question type classification) to route queries. Faster but less flexible. Hybrid Routing: Combine rules for obvious cases with LLM-based routing for ambiguous queries. Multi-Source Routing: Direct queries to different data sources based on the topic. For example: • Technical questions → documentation vector store • Policy questions → policy database • Current events → web search The routing mechanism should also handle edge cases: • Questions that don't need retrieval (general knowledge, greetings) • Questions that require multiple data sources • Questions that are out of scope for the system

Learning Objectives

What is Agentic RAG?

Agent Architecture

Implementation with LangGraph

Routing Strategies

Key Takeaways