Self-RAG

What is Self-RAG?

Self-RAG is an approach where the language model itself learns to decide when to retrieve, evaluates whether retrieved evidence supports its claims, and assesses the quality of its own generated response. It introduces a self-reflective loop into the RAG process. Example Scenario: A technical assistant receives a question about a complex topic. Instead of blindly retrieving and generating, Self-RAG: 1. Decides whether retrieval is needed for this particular question 2. Retrieves documents and evaluates whether they are relevant 3. Generates a response segment 4. Checks whether the generated text is supported by the evidence 5. Evaluates overall response quality 6. May regenerate or adjust based on self-evaluation The key insight is that not every question requires retrieval, and not every retrieved document is useful. Self-RAG makes these decisions explicitly.

Full Self-RAG vs Approximations

The original Self-RAG paper (Asai et al., 2023) trains the model to generate special reflection tokens. A simplified version uses separate LLM calls to approximate these evaluations. The simplified version is useful but doesn't capture all the benefits of the trained approach.

Self-Reflection Mechanisms

Self-RAG introduces four types of self-reflection: 1. Retrieve Decision — Should I retrieve documents for this question? • Some questions can be answered from the model's knowledge (e.g., "What year did World War II end?") • Complex or domain-specific questions benefit from retrieval • The model outputs a decision: "retrieve" or "no_retrieve" 2. Relevance Assessment (ISREL) — Are the retrieved documents relevant? • For each retrieved document, assess whether it contains information relevant to the query • Documents rated as irrelevant can be discarded • Similar to CRAG's evaluation step 3. Support Assessment (ISSUP) — Does the evidence support the generated claim? • After generating a response segment, check whether it is supported by the retrieved evidence • Categories: "fully supported", "partially supported", "not supported" • Unsupported claims indicate potential hallucination 4. Utility Assessment (ISUSE) — Is the overall response useful? • Evaluate the complete generated response for relevance, completeness, and quality • Scores typically range from 1-5 • Low utility triggers regeneration

Simplified Self-RAG Implementation

A practical approximation of Self-RAG uses separate LLM calls to perform the reflection steps. While not as efficient as the trained approach, it demonstrates the core concepts:

Note

This simplified implementation uses 4-6 LLM calls per query, which adds latency and cost. The original Self-RAG trains a single model to generate reflection tokens inline, making it much more efficient.

self_rag.pypython

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4", temperature=0)

def should_retrieve(question: str) -> bool:
    """Decide if retrieval is needed."""
    response = llm.invoke(
        f"""Does this question require looking up specific documents 
        or information to answer accurately? 
        Question: {question}
        Answer with 'yes' or 'no'."""
    )
    return "yes" in response.content.lower()

def check_relevance(question: str, document: str) -> bool:
    """Check if a document is relevant (ISREL)."""
    response = llm.invoke(
        f"""Is this document relevant to answering the question?
        Question: {question}
        Document: {document}
        Answer 'relevant' or 'irrelevant'."""
    )
    return "relevant" in response.content.lower()

def check_support(claim: str, evidence: str) -> str:
    """Check if evidence supports a claim (ISSUP)."""
    response = llm.invoke(
        f"""Does the evidence support this claim?
        Claim: {claim}
        Evidence: {evidence}
        Answer: 'fully_supported', 'partially_supported', or 'not_supported'."""
    )
    return response.content.strip().lower()

def check_utility(question: str, answer: str) -> int:
    """Rate the utility of the answer (ISUSE), 1-5."""
    response = llm.invoke(
        f"""Rate how well this answer addresses the question (1-5).
        Question: {question}
        Answer: {answer}
        Rating (just the number):"""
    )
    try:
        return int(response.content.strip())
    except ValueError:
        return 3

def self_rag_pipeline(question: str) -> str:
    # Step 1: Decide if retrieval is needed
    if should_retrieve(question):
        docs = retriever.invoke(question)
        
        # Step 2: Filter relevant documents
        relevant_docs = [d for d in docs if check_relevance(question, d.page_content)]
        
        if not relevant_docs:
            return "I don't have sufficient information to answer this question."
        
        context = "\n".join(d.page_content for d in relevant_docs)
    else:
        context = ""
    
    # Step 3: Generate answer
    prompt = f"Context: {context}\n\nQuestion: {question}\n\nAnswer:" if context else f"Question: {question}\n\nAnswer:"
    answer = llm.invoke(prompt).content
    
    # Step 4: Check support and utility
    if context:
        support = check_support(answer, context)
        if support == "not_supported":
            # Regenerate with stricter grounding instruction
            answer = llm.invoke(
                f"Based ONLY on this context, answer the question. "
                f"If the context doesn't contain the answer, say so.\n"
                f"Context: {context}\nQuestion: {question}\nAnswer:"
            ).content
    
    utility = check_utility(question, answer)
    if utility <= 2:
        answer += "\n\n(Note: This answer may be incomplete. Consider refining your question.)"
    
    return answer

Learning Objectives

What is Self-RAG?

Self-Reflection Mechanisms

Simplified Self-RAG Implementation

Key Takeaways