IntermediateEstimated time: 35 mins

100% Local RAG with Ollama & Chroma

Run a completely private, offline RAG system with zero external API calls or subscription costs.

Tech Stack:Ollama (Llama 3.2 / Mistral)nomic-embed-textChromaDBPythonLangChain

System Architecture

Local PDF ➔ nomic-embed-text (Local Ollama) ➔ ChromaDB (Local SQLite/Parquet) ➔ Llama-3.2 (Local GPU/CPU) ➔ Completely Private Answer

Prerequisites

Ollama installed from ollama.com
Downloaded models: ollama pull llama3.2 && ollama pull nomic-embed-text
8GB+ RAM (16GB recommended for 7B/8B models)

Installation Commands

terminalbash

ollama pull llama3.2
ollama pull nomic-embed-text
pip install langchain langchain-community chromadb

Folder Structure

Directory Layouttext

local-rag/
├── local_rag.py
└── data/
    └── company_policy.txt

Step-by-Step Implementation

Local Embeddings and Model Setup

Initialize LangChain using local Ollama endpoints on localhost:11434.

local_rag.pypython

from langchain_community.llms import Ollama
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import PromptTemplate
from langchain.chains import RetrievalQA

# 1. Local Embeddings via Ollama
embeddings = OllamaEmbeddings(model="nomic-embed-text")

# 2. Local In-Memory ChromaDB
texts = [
    "ACME Corp offers 25 days paid annual leave for all engineers.",
    "Travel expenses under $50 do not require itemized receipts.",
]
vectorstore = Chroma.from_texts(texts, embeddings)

# 3. Local LLM via Ollama
llm = Ollama(model="llama3.2", temperature=0.1)

# 4. Local QA Chain
qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 1})
)

query = "How much annual leave do engineers get at ACME?"
response = qa.invoke(query)
print("Answer:", response["result"])

Example Execution (Input & Output)

Input Query:

"How much annual leave do engineers get at ACME?"

Generated Grounded Output:

"At ACME Corp, engineers receive 25 days of paid annual leave."

Why this works: Both the embeddings calculation and inference occurred locally on device without transmitting data over the internet.

Troubleshooting Common Gotchas

Connection error: Failed to connect to localhost:11434

Make sure the Ollama background service is running by executing "ollama serve" in a separate terminal.

Known Limitations

•Inference speed depends on local hardware (CPU vs GPU with VRAM).

Security Considerations

•Ideal for HIPAA, GDPR, and confidential trade-secret document analysis.

Back to all projects