Production RAG
From Scratch to Scalable Systems. Retrieval-augmented generation that survives contact with a real corpus, real users, a real bill and a real adversary — stage by stage, with the code and the numbers.
- 16
- parts
- 379
- sections
- 331
- code listings
- 134k
- words
Read it front to back once to build the mental model, then use the table of contents as a lookup index. Every part ends with a checklist you can act on.
The shape of the problem. A RAG system is two pipelines that meet at a vector store. The indexing pipeline runs offline — load, split, embed, store. The retrieval pipeline runs per query — embed the question, search, rank, stuff a prompt, generate. Nearly every production failure traces to a specific stage of one of those two pipelines, so the guide is organised the same way: build the pipelines, then make each stage observable, cheap, fast and safe.
Two pipelines that meet at a vector store
Every part of the guide owns one or more boxes of this map. Read it once; then use it as the index to everything else.
Sixteen parts, four tracks
Read it front to back once to build the mental model, then use it as a lookup index. Every part ends with a checklist you can act on.
Foundations
Build the two pipelines
Why RAG Breaks in Production, and How the Pieces Fit
Name the failure modes, install the end-to-end map, the canonical chain and the six invariants.
Set Up the Workshop and Get Data In
Project setup with uv, credentials, and the loader layer that turns any file into Documents.
Chunking: The Single Biggest Quality Lever
Splitters, chunk size and overlap, semantic chunking and an A/B harness: the lever that decides everything downstream.
Embeddings, Vector Stores and Reading Similarity Scores
Turn chunks into vectors, store them in Chroma, and learn to read similarity scores correctly.
Your First Real RAG System, and Debugging It When It Lies
Wire the first end-to-end chain with LCEL, then debug it systematically when it answers wrong.
Quality & operations
Make it right, visible and affordable
Retrieval Quality: Hybrid Search, Reranking and Citations
Hybrid BM25 + vector search fused with RRF, reranking, compression and citations.
Observability: Traces, Metrics and Evals
Tracing, metrics and evals: the frame around the pipeline that changes nothing and reveals everything.
Scaling: Index Tuning, HNSW and When to Shard
HNSW's M and ef, recall vs latency presets, vertical scaling first and when to shard.
The Cost and Latency Budget
Quantization, batching, three caching layers and a worked cost model: get the money back.
Shipping to production
Host it, serve it, defend it, test it
Production Hosting: Supabase and pgvector
From a local folder to Supabase + pgvector: managed vs self-hosted, the walkthrough, the operations.
The Production API: FastAPI and the Request Path
A FastAPI + LangGraph service: rate limits, cache, retries, fallbacks, circuit breaker, metrics.
The Security Layer: Prompt Injection and PII
Prompt-injection screening, PII masking and output validation in about a hundred lines.
Testing the Security Layer, and the Production Checklist
Adversarial tests, LLM-as-judge evals, the Docker image and the production checklist.
Advanced & frontier
Change what retrieval means
Advanced Retrieval: Long Context, Contextual Retrieval and Late Chunking
Long context vs RAG, contextual retrieval and late chunking: fix what each chunk means.
Agentic RAG: Self-Correcting Retrieval with LangGraph
Self-correcting retrieval: grade, rewrite and retry with a LangGraph state machine.
The Frontier: GraphRAG, Multimodal RAG and Where This Is Going
GraphRAG and multimodal ColPali retrieval: when a document is not just text.
Audio summaries
Short spoken overviews of the parts that benefit most from being explained out loud. Generated with NotebookLM from the text of each part.
- 01Why RAG Breaks in Production, and How the Pieces Fit5:51
- 03Chunking: The Single Biggest Quality Lever5:27
- 04Embeddings, Vector Stores and Reading Similarity Scores7:02
- 05Your First Real RAG System, and Debugging It When It Lies5:21
- 06Retrieval Quality: Hybrid Search, Reranking and Citations5:54
- 07Observability: Traces, Metrics and Evals6:20
- 08Scaling: Index Tuning, HNSW and When to Shard4:33
- 09The Cost and Latency Budget6:14
- 11The Production API: FastAPI and the Request Path5:40
- 12The Security Layer: Prompt Injection and PII6:13
- 13Testing the Security Layer, and the Production Checklist5:55
- 14Advanced Retrieval: Long Context, Contextual Retrieval and Late Chunking4:36
- 15Agentic RAG: Self-Correcting Retrieval with LangGraph6:02
- 16The Frontier: GraphRAG, Multimodal RAG and Where This Is Going5:13