Multimodal RAG

What is Multimodal RAG?

Multimodal RAG extends the RAG paradigm beyond text to handle documents containing multiple types of content: text, images, tables, charts, diagrams, and other visual elements. Real-world documents rarely contain only text — PDFs include figures, tables, and charts that carry important information. Example Scenario: A user asks a question about a PDF that contains: • Text paragraphs explaining a concept • A data table comparing different options • A chart visualizing performance metrics • A diagram showing system architecture A text-only RAG system would miss information encoded in non-text elements. Multimodal RAG processes all these modalities and retrieves the most relevant pieces regardless of type.

Implementation Complexity

Multimodal RAG is significantly more complex than text-only RAG. PDF parsing, OCR, vision models, and indexing strategies depend heavily on the document format and chosen implementation. Start with text-only RAG and add modalities incrementally.

Multimodal Document Parsing

Extracting structured content from complex documents is the foundation of multimodal RAG: Text Extraction: Standard text extraction from PDFs using libraries like PyPDF2, pdfplumber, or unstructured. Table Extraction: Tables require specialized parsing to preserve their structure: • pdfplumber: Extracts table data as structured arrays • Camelot: Specialized PDF table extraction • Unstructured: General-purpose document parsing with table detection • Vision models: Use GPT-4V or similar models to interpret table screenshots Image Extraction: Extract embedded images from documents: • PyMuPDF (fitz): Extract images from PDF pages • pdf2image: Convert PDF pages to images for vision model processing Chart/Diagram Interpretation: Charts and diagrams require vision models: • GPT-4V: Describe and interpret chart content • Open-source vision models for on-premise processing Layout Analysis: Understanding the spatial layout of a document page helps associate figures with their captions and relevant text sections.

multimodal_parsing.pypython

from unstructured.partition.pdf import partition_pdf

# Parse PDF with table and image extraction
elements = partition_pdf(
    filename="report.pdf",
    strategy="hi_res",           # High-resolution parsing
    extract_images_in_pdf=True,  # Extract embedded images
    infer_table_structure=True,  # Detect and parse tables
)

# Categorize extracted elements
texts = []
tables = []
images = []

for element in elements:
    if element.category == "Table":
        tables.append(element.metadata.text_as_html)  # HTML table
    elif element.category == "Image":
        images.append(element.metadata.image_path)     # Saved image path
    else:
        texts.append(element.text)                     # Text content

print(f"Extracted: {len(texts)} text blocks, {len(tables)} tables, {len(images)} images")

Cross-Modal Retrieval

In multimodal RAG, retrieval must work across different content types. There are several strategies: Strategy 1: Text Summaries for All Modalities Convert non-text elements to text descriptions, then use standard text embedding and retrieval: • Tables → text descriptions of key data points • Images → captions generated by vision models • Charts → textual summaries of trends and values This is the simplest approach and works with standard vector databases. Strategy 2: Multimodal Embeddings Use embedding models that can handle multiple modalities (like CLIP for text and images) to embed all content types into a shared vector space. This enables direct cross-modal similarity search. Strategy 3: Separate Indices Maintain separate indices for each modality and merge results: • Text index: standard text embeddings • Image index: image embeddings or caption embeddings • Table index: structured data or table description embeddings Merge results using methods like RRF.

multimodal_retrieval.pypython

from langchain_openai import ChatOpenAI

vision_llm = ChatOpenAI(model="gpt-4o", temperature=0)

def summarize_table(table_html: str) -> str:
    """Generate a text summary of a table for embedding."""
    response = vision_llm.invoke(
        f"Summarize the key information in this HTML table in 2-3 sentences. "
        f"Include specific numbers and comparisons:\n{table_html}"
    )
    return response.content

def describe_image(image_path: str) -> str:
    """Generate a text description of an image for embedding."""
    import base64
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()
    
    response = vision_llm.invoke([
        {"type": "text", "text": "Describe this image in detail, including any text, data, or diagrams shown."},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}}
    ])
    return response.content

# Create unified text representations
all_chunks = []
for text in texts:
    all_chunks.append({"content": text, "type": "text"})
for table in tables:
    summary = summarize_table(table)
    all_chunks.append({"content": summary, "type": "table", "raw": table})
for img_path in images:
    description = describe_image(img_path)
    all_chunks.append({"content": description, "type": "image", "path": img_path})

# Now embed and store all chunks uniformly
# Retrieval works the same as text-only RAG

Combining Multimodal Evidence

When generating the final answer, the LLM needs context from multiple modalities. How you present this context matters: Text Context: Include directly in the prompt as usual. Table Context: Include the original HTML/markdown table along with its summary. LLMs can interpret tabular data when it's well-formatted. Image Context: For text-based LLMs, include the generated description. For vision-capable LLMs (like GPT-4o), you can include the actual image alongside text. Practical Considerations: • Vision model API calls add latency and cost • Image quality affects vision model accuracy • OCR may be needed for scanned documents • Table structure preservation is critical for accuracy • Not all PDFs are created equal — some are well-structured, others are scanned images

Learning Objectives

What is Multimodal RAG?

Multimodal Document Parsing

Combining Multimodal Evidence

Key Takeaways