oriolrius.me Production RAGReference guide
Part 01·Foundations

Why RAG Breaks in Production, and How the Pieces Fit

Name the failure modes, install the end-to-end map, the canonical chain and the six invariants.

37 min read 25 sections Foundations
Audio summary · 5:51 · EN
Why RAG Breaks in Production, and How the Pieces Fit

You have built RAG before. It worked. You demoed it, the answers were good, and then you pointed it at the real corpus and it stopped being good — without ever throwing an exception, without a single red line in the logs, without a metric moving. That specific experience is the entire premise of this guide, and it is where the source course opens too 00:00:12. Its stated goal is the transition from prototype to production, not an introduction to RAG — and that distinction sets everything that follows. You are not being taught what an embedding is; you are being taught why yours is not being retrieved.

This part does two jobs. First it names the failures precisely enough that every later part can point at one and say this is the box I am fixing. Second it installs the map — the end-to-end picture, the canonical chain that the remaining fifteen parts only ever swap one component of, and the six invariants that everything else obeys. Read this part once linearly. After that, use it as the index you come back to when something breaks.

The cliff: ten documents to ten thousand

"It worked on 10 documents, but then you decide to add 10,000 documents and everything broke." 00:00:12

The detail that matters in that sentence is the one that is missing: the code did not change. Same loader, same splitter, same embedding model, same vector store, same prompt. The only variable that moved was corpus size, by a factor of a thousand, and the system went from useful to useless.

The instructor's follow-up aside — "sound familiar?" 00:00:19 — is doing real work too. Hitting this wall is the normal outcome of following a tutorial, not evidence that you did something stupid. The wall is a property of the architecture, not of your discipline.

And it is not a bug in the sense you are used to. It is a property of how RAG degrades. A ten-document corpus hides every defect in your pipeline because nothing competes for the top-k slots. The demo knowledge base you will meet later in this part is a 1,010-character Markdown string split at chunk_size=500; the demo then retrieves with k=2. Whatever the splitter did, a large fraction of the whole corpus lands in the prompt on every question. The right answer is in there by accident. Retrieval quality is not being tested; it is being bypassed.

At ten thousand documents — easily hundreds of thousands of chunks — every one of those defects becomes decisive. Now there are thousands of chunks that are plausibly about your query, and the ranking function has to actually be right. A chunk split mid-sentence now loses to a competitor that wasn't. A query phrased in the user's vocabulary now loses to documents phrased in the company's vocabulary. Nothing errors. The system just quietly starts answering from the wrong two chunks.

The 90% claim, and the half of it worth keeping

"90% of [RAG] systems out there, they fail in production." 00:00:22 "And they all fail for the same reasons." 00:00:24

Be honest about what this is. It is the instructor's field estimate from teaching, stated flatly with no citation, and the part-6 README rephrases it as *"why 90% of RAG projects fail in production"* — projects in the written source, systems in the audio. Same claim, two nouns, no study behind either. Do not carry it into a design document as a fact.

The second clause is the one that earns its place. If RAG failures were idiosyncratic — if every broken system broke in its own novel way — then no curriculum, checklist or hardening process could exist. The reason a guide like this can be written at all is that the failure set is small, finite and enumerable. The course's own framing (the part-6 README credits the taxonomy to teaching 300K+ students and seeing the same problems repeatedly) puts the count at five. Whether the real number is five or seven is not the point; the point is that it is bounded, and a bounded failure set can be turned into a diagnostic procedure.

Who this is for, and what is assumed

The instructor lets it slip in an aside at 00:00:29 that the assumed learner has already built RAG systems that don't work. That sets the level precisely, and this guide inherits it:

RequirementLevel
PythonComfortable with functions, classes, package management
APIsBasic understanding of REST
LLMsHelpful, not required
ML / AI backgroundNot required
GPUNot required — all code runs on CPU; the one exception (ColPali, Part 16) is optional

You are not a beginner at building. You are a beginner at making it survive contact with a real corpus, real users, a real bill and a real adversary. This guide therefore does not open with "what is an embedding".

One more thing is worth noticing about the source's framing before we start naming things: the spoken introduction never mentions a vendor, a model or a library. Not OpenAI, not LangChain, not Chroma — those appear only in the video's title. The failure modes are stated as properties of RAG, not of a stack. That is why the diagnoses in this part transfer even if you swap every component: the fixes are architectural, and only the code samples are LangChain-flavoured.

Provenance and trust rules for everything that follows

Everything in this guide is derived from one source with known defects, and you should know its shape before you trust a number in it.

PropertyValue
SourcefreeCodeCamp, Production RAG with LangChain & Vector Databases – Full Course, uploaded 2026-05-26
InstructorPaulo Dichone (350,000+ students, 70+ courses, building production AI since 2015 — per the part-6 README, not spoken)
Runtime07:38:38 across 32 chapters
Transcript10,598 caption lines · 73,539 words, auto-generated
Video1.26 GB p.webm, AV1, 1920×1080, Opus audio
Companion repo Aproduction-course-main-code — requires-python = ">=3.12", project langc-course; its README.md is empty (0 bytes)
Companion repo Bfcc-production-rag-part-6 — requires-python = ">=3.13", project fcc-production-rag, requirements header dated "April 2026"; the only written README in either repo

The two chapters this part is built from are the two shortest teaching chapters in the course — chapter 1 is 1 min 44 s / 276 words of pure framing with no code and no demo, and chapter 2 is 6 min 43 s / 1,133 words of slide walkthrough, also with no live coding. Everything executable below is the repository equivalent of what the slides depicted, opened and read rather than reconstructed from the narration. Nothing in this part was typed on screen by the instructor, and nothing in it was reconstructed from audio.

The exact dependency pins, both repos

production-course-main-code/pyproject.toml (requires-python = ">=3.12", project name langc-course):

TOML
dependencies = [
    "langchain-community>=0.4.1",
    "langchain>=1.2.7",
    "langchain-anthropic>=1.3.1",
    "langchain-core>=1.2.7",
    "langchain-openai>=1.1.7",
    "langgraph>=1.0.7",
    "python-dotenv>=1.2.1",
    "bs4>=0.0.2",
    "pypdf>=6.6.2",
    "langchain-ollama>=1.0.1",
    "langchain-chroma>=1.1.0",
    "rank-bm25>=0.2.2",
    "langgraph-checkpoint-sqlite>=3.0.3",
    "pytest>=9.0.2",
    "fastapi>=0.129.0",
    "uvicorn>=0.40.0",
]

fcc-production-rag-part-6/pyproject.toml (requires-python = ">=3.13", project name fcc-production-rag):

TOML
dependencies = [
    "langchain>=1.2.0,<2.0.0",
    "langchain-chroma>=1.1.0,<2.0.0",
    "langchain-openai>=1.1.0,<2.0.0",
    "langchain-text-splitters>=1.1.2",
    "networkx>=3.6.1",
    "openai>=2.32.0",
    "python-dotenv>=1.0.0",
    "tiktoken>=0.12.0",
]

The part-6 requirements.txt additionally pins langgraph>=1.1.0, chromadb>=1.5.0, langchain-classic>=1.0.0, pypdf>=6.0.0, pyvis>=0.3.2, pillow>=11.0.0, sentence-transformers>=5.0.0 and langchain-anthropic>=0.4.0 — the graph, vision and local-embedding dependencies that only Part 16 needs. Two repos means two virtual environments; Part 16's material genuinely requires the second one.

Four concrete contradictions you will hit if you clone the repos:

  • The README claims Python 3.10+. The actual pyproject.toml files pin ≥3.12 and ≥3.13. Trust the code.
  • The README lists Streamlit in the tech stack. It appears in neither dependency file; the main repo ships FastAPI + uvicorn instead.
  • The README's git clone https://github.com/pdichone/production-rag-course.git and its code/part1-foundation/ layout do not match what ships — the part-6 checkout is flat at the repo root and its pyproject.toml declares name = "fcc-production-rag".
  • The README's step 3 is cp .env.example .env. Neither checkout contains a .env.example. You write the .env from scratch; Part 2 tells you what goes in it. (Both repos do ship a uv.lock, so uv — not the README's python -m venv + pip — is how this code was actually built. This guide uses uv throughout.)

The build order is a rule, not a syllabus

Four verbs, in this order 00:00:35:

  1. Build — get a pipeline that runs end to end.
  2. Debug — find out why it returns the wrong answer.
  3. Optimize — fix retrieval quality.
  4. Scale — only now, make it fast and cheap at volume.

"We're going to optimize for quality, then scale for production." 00:01:00

The tutorial ecosystem stops at verb one. The ordering of the other three is prescriptive, not stylistic. You cannot optimize what you cannot observe, which is why this guide moves observability earlier than the course does. And you must not scale what you have not optimized, for a reason that is worth stating bluntly:

Scaling a low-quality retriever produces wrong answers faster, at higher volume, and at greater cost. Scale multiplies whatever you already have — including the wrongness.

Combine the course blurb's three "challenges standard tutorials ignore" — scaling, debugging, security — with the roadmap's "optimize for quality", and you get the four axes this guide's production checklist is organised around:

AxisThe question it answersOwned by
QualityDoes it retrieve and answer correctly?Parts 3–6
ScaleDoes it stay correct, fast and affordable at volume?Parts 8–10
Observability / debuggabilityWhen it is wrong, can you find out why?Parts 7, 13
SecurityDoes it survive a hostile user?Parts 12, 13

The source course votes with its runtime on which of those it thinks is hardest: the security block is roughly 67 minutes, the largest single block in a 7½-hour course, and the optimization chapter at 35 minutes is the longest individual technical chapter. Foundations — everything most tutorials consist of — get about ninety minutes of the seven and a half hours.

The course's own acceptance criteria for "ready for deployment" are robust, secure, ready — note that fast and cheap are deliberately not the headline. Correctness under load and safety are.

The source's six parts, and this guide's sixteen

The course announces a six-part arc between 00:00:44 and 00:01:44. This guide reorganises it into sixteen parts, in build order, but the arc is the same and it is worth seeing the mapping so you can find the original material when you want it.

Course partTopicIts deliverableThis guide
1Build the FoundationA complete RAG pipeline from scratchParts 2–5
2Debug RAG FailuresFix the 5 failure modesParts 3, 5, 7
3Optimize for QualitySemantic chunking, reranking, multi-query retrievalParts 3, 6, 9
4Scale for ProductionCaching, monitoring, production vector databasesParts 7–10
5Production ProjectA full production-ready RAG applicationParts 11–13
6Advanced RAG 2026Agentic RAG, GraphRAG, Contextual Retrieval, MultimodalParts 14–16

Course part 1 names four foundation primitives in pipeline order — documents, chunks, embeddings, vector stores 00:00:44 — and that ordering is itself the mental model of the indexing phase: a document is loaded, split into chunks, each chunk is embedded, the vectors land in a store. Parts 2–4 of this guide execute exactly that sequence.

Course part 6 names four cutting-edge techniques — agentic RAG 00:01:22, GraphRAG, contextual retrieval and multimodal RAG 00:01:44 — but the repository ships six lesson files: long-context-vs-RAG and late chunking are taught without being advertised. All six are covered in Parts 14–16.

The five failure modes

This is the taxonomy the rest of the guide points back at. The instructor alludes to it between 00:00:52 and 00:01:00 — "the five main failure modes" — but never enumerates them on air; the enumeration lives in the part-6 README, and it is reproduced here with the guide part that owns each fix.

#Failure modeWhat happensThe fixOwned by
1Bad chunkingWrong context retrieved; chunks split mid-sentenceSemantic chunking, smart overlapPart 3
2Embedding mismatchUser says "cancel", docs say "termination policy"Query rewriting, hybrid searchParts 4–6
3Retrieval noise10 docs retrieved, only 2 are relevantReranking, filteringPart 6
4Context overflowToo much stuffed in the prompt, the LLM ignores halfSmart truncation, map-reducePart 9
5HallucinationThe answer is in the context and the LLM makes things up anywayConstrained prompts, citationsThis part; detected in Part 7, tested in Part 13, self-graded in Part 15

Read them once carefully, because the phrasing is doing work.

1. Bad chunking. The splitter cuts through a sentence, a code block, a table row or — worst — the middle of a procedure. The embedding model then faithfully encodes the fragment. Nothing is broken in any component; the vector is a perfectly accurate representation of an incomplete thought. Part 3 shows this with a 50-page API document where the OAuth2 answer straddles chunks 47 and 48, and neither half is retrievable as an answer.

2. Embedding mismatch. The user's vocabulary and the corpus's vocabulary are different dialects. "Cancel" and "termination policy" mean the same thing to a human and land in different neighbourhoods of vector space. Pure dense retrieval has no lexical anchor to rescue it. The fixes are rewriting the query into the corpus's dialect and adding a keyword channel (BM25) that hybrid-fuses with the dense channel.

3. Retrieval noise. You asked for ten chunks and two of them are relevant. The other eight are not merely useless — they are active distractors that consume context budget and give the model plausible-looking wrong material to synthesise from. The fix is a second-stage reranker and metadata filtering.

4. Context overflow. You solved noise by retrieving more, and now the prompt is huge and the model is skipping the middle of it. More context is not more grounding past a point. The fixes are truncation that is aware of what it is throwing away, and map-reduce strategies that summarise before they synthesise.

5. Hallucination. This is the one that surprises people, because of the precise wording: the answer is in the context and the model invents anyway. This is not a retrieval failure. Retrieval did its job. The model was simply never told it was forbidden to go beyond what it was given. The fix is a constrained prompt and citations — and it is the cheapest fix in the whole taxonomy, which is why it is taught in this part rather than deferred.

The end-to-end picture

Here is the whole system, in the order data flows through it at query time 00:01:44:

Figure 1The basic RAG chain: retrieve context, fill the prompt, generate a grounded answer
Original ASCII
user query ──► retriever ──► vector store ──► context
     │                                          │
     └──────── question ────────────────────────┤
                                                ▼
                                         prompt template
                                                │
                                                ▼
                                              LLM
                                                │
                                                ▼
                                     grounded response (+ citations)

Five observations about that picture are load-bearing later.

The query plays two roles at once. It is both the thing the user wants answered and the retrieval key — which is why it appears twice in the diagram and only once in the input. One string, two consumers. This dual role is exactly why the question must survive the pipeline unmodified; more on this when we get to RunnablePassthrough.

Three things converge on the prompt template, and only one thing leaves it. The question, the retrieved context and the template's own instructions merge into a single filled string. That string is the only artefact the model ever sees. Everything else in the diagram — the retriever, the store, the metadata, your careful scoring — exists solely to decide what goes into it. If a fact is not in that string, the model does not have it; if a citation marker is not in that string, the model cannot cite it.

The retriever is a thin façade; the vector store is the engine.

"all of that of course is driven by the vector store" 00:01:44

A retriever is a small object with an invoke(question) -> list[Document] contract. It holds essentially no intelligence. Every property you care about — recall, latency, filter support, hybrid capability — is a property of the store and its index underneath. When retrieval is bad, do not go looking in the retriever.

Embedding is not indexing. 00:02:46 The instructor describes the store as holding documents that are embedded and indexed for search, and that "and" is two separate properties, not a restatement:

  • Embedding turns text into vectors. It is a model choice: which model, how many dimensions, what it costs per million tokens.
  • Indexing builds the search structure over those vectors — HNSW, IVF. It is an infrastructure choice with its own parameters (M, ef_construction, ef_search) and its own accuracy/speed/memory trade-off.

You can have excellent embeddings inside a badly tuned index and get slow, low-recall search. You can have a beautifully tuned HNSW graph over the wrong embedding model and get fast, confident nonsense. Conflating the two is why people "upgrade the embedding model" to fix a latency problem. This single sentence is the seed of the entire index-tuning arc in Part 8.

The output is *grounded*, and grounding is a mitigation, not a cure.

"RAG grounds large language model responses in actual documents which reduces hallucination" 00:02:46

The verb is reduces. Not eliminates, not prevents, not solves. Any vendor page, blog post or architecture review that claims RAG eliminates hallucination is wrong, and the rest of this guide — debugging, evaluation, output validation, security — exists precisely because the residual is large enough to need engineering.

The canonical chain

Everything above becomes about ten lines of LCEL. This is the shape you will see in every part of this guide, and — this is the important claim — the only thing that ever changes is what sits in the context slot.

Python
# production-course-main-code/rag_pipeline.py — demo_basic_rag()
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Four stages: {context, question} → prompt template → LLM → output parser 00:02:46. Take them one at a time.

Before we do, look at where those names come from. It is worth seeing the import block once, because everything in Parts 2–6 draws from the same set and beginners lose hours to importing the right class from the wrong package. This is the head of rag_pipeline.py, unedited:

Python
# production-course-main-code/rag_pipeline.py
from langchain_openai.embeddings import OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.chat_models import init_chat_model

from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from pydantic import BaseModel, Field
from typing import List
from dotenv import load_dotenv
import tempfile

Read that block as a map of the ecosystem: langchain_core owns the composable primitives (prompts, runnables, parsers, Document), langchain owns the provider-agnostic factory init_chat_model, langchain_openai owns the OpenAI bindings, langchain_chroma owns the store, and langchain_text_splitters is its own package because splitting has no dependencies on any of the rest. Chroma comes from langchain_chroma, not langchain_community — the trap flagged above.

It also contains three artefacts worth naming, because you will produce the same ones: OpenAIEmbeddings is imported twice from two paths, ChatOpenAI is imported and never used (the file goes through init_chat_model instead), and RunnableParallel is imported and never used (the dict literal covers it). None of that breaks anything; all of it is what a real teaching repo looks like. Trust the code over the README, but read the code critically too.

Stage 1: the dict literal is an implicit RunnableParallel

"This will come in parallel inputs of course." 00:02:46

That {"context": ..., "question": ...} is not a Python dict being passed as an argument. Inside an LCEL pipe, a dict literal is coerced into a RunnableParallel: each value is a runnable, they all receive the same input, and they execute side by side. The output is a dict with the same keys and the branches' results as values.

This matters mechanically and conceptually. Mechanically, retrieval does not happen "before" the question is handled — both branches consume the same input string concurrently and merge into one dict, which is what the prompt template consumes. Conceptually, it puts the whole acronym on screen. The context branch is Retrieval. The moment the two branches merge into one dict is Augmentation — the instructor points at exactly this step and calls it "part of augmented" 00:03:47. The | llm stage is Generation. Three letters, three positions in one expression; if you can point at each of them in the code you understand what RAG is doing, and if you cannot, no amount of tuning will help you.

"Context and question are going to go hand in hand." 00:02:46

If you prefer the explicit form — and you should when you are learning, or when you need to name the branches for tracing — chains_v1.py writes the same thing out longhand with a stub retriever, which isolates the passthrough concept from any vector store. (The instructor introduces RunnablePassthrough with "If you remember" 00:03:47 — this is a production course, not LangChain 101, and prior LCEL familiarity is assumed. This guide makes the same assumption, so here is the refresher he skips.) Note the model there is built with init_chat_model(model="gpt-4o-mini", temperature=0) — zero, not the 0.2 used in the RAG demos, because chain-mechanics demos want determinism.

Python
# production-course-main-code/chains_v1.py — demo_passthrough_chain()
prompt = ChatPromptTemplate.from_template(
    "Original question: {question}\n"
    "Context: {context}\n\n"
    "Answer the question based on the context."
)

# simulate a retrieve operation
def fake_retriever(input_dict):
    return " LangChain was created by Harrison Chase in 2022."

chain = (
    RunnableParallel(
        context=RunnableLambda(fake_retriever), question=RunnablePassthrough()
    )
    | RunnableLambda(
        lambda x: {"context": x["context"], "question": x["question"]["question"]}
    )
    | prompt
    | model
    | StrOutputParser()
)

result = chain.invoke({"question": "Who created LangChain?"})

Note the extra RunnableLambda in the middle. Because this chain is invoked with a dict rather than a bare string, RunnablePassthrough() forwards the whole dict, so the question key holds {"question": "..."} and has to be unwrapped before the prompt sees it. In rag_pipeline.py the chain is invoked with a bare string (rag_chain.invoke(q)), so the passthrough yields the string directly and no unwrapping step is needed. Same operator, different input type, different plumbing — this is the single most common LCEL papercut.

Stage 1b: RunnablePassthrough means "do not touch this"

"do not modify anything when it comes to the question" 00:03:47

Mechanically it forwards its input unchanged. Semantically it is a contract, and the semantic reading is the one worth internalising: the user's exact wording must reach the prompt. If you "helpfully" clean up, expand, translate or summarise the question in this branch, you have silently changed the question the model is answering, and your evaluation harness will be scoring answers to a question the user never asked.

Stage 2: the prompt template carries information and instructions

Two placeholders, two provenances 00:04:49: {context} comes from the retriever branch, {question} from the passthrough branch. But the template is not just a string formatter — it is where you ground the model, which means it carries both the information (the retrieved chunks) and the instructions (how to behave with them). The next section is entirely about those instructions.

Stage 3 and 4: the model, and the parser you should not skip

The model is gpt-4o-mini at temperature=0.2 in rag_pipeline.py, built through the provider-agnostic factory:

Python
# production-course-main-code/rag_pipeline.py
llm = init_chat_model(model="gpt-4o-mini", temperature=0.2)
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")

init_chat_model is the provider-agnostic factory: the model identifier is a string argument, so swapping OpenAI for Anthropic is a one-token edit and the chain around it does not move. That is invariant 2's asymmetry made concrete — the generation model is swappable, the embedding model on the line below it is not. Note also that this repo runs its RAG demos at temperature=0.2 and its chain-mechanics demos at temperature=0; pick one deliberately and write it down, because a non-zero temperature makes your evaluation runs non-reproducible.

.invoke() on a chat model returns a message object, not a string — the text lives on .content, alongside token usage and stop reason. StrOutputParser() is the one-line adaptor that unwraps it, and the instructor's phrasing about output parsers is "most cases we need this" 00:02:46. The richer reason to keep the slot is that it is your control point: swap StrOutputParser for a structured parser and you get a typed object instead of prose. rag_pipeline.py demonstrates exactly that, and it is worth seeing now because it is the seed of the output-validation layer in Part 11:

Structured output — the same chain with a typed contract instead of a string
Python
# production-course-main-code/rag_pipeline.py — demo_structured_rag()
# (in the repo every name below is defined inside the demo function; the body is verbatim)
vectorstore = create_kb()
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

class RAGResponse(BaseModel):
    """Structured RAG response."""

    answer: str = Field(description="The answer to the question")
    confidence: str = Field(description="high, medium, or low")
    sources_used: List[str] = Field(description="List of sources referenced")
    follow_up: str = Field(description="Suggested follow-up question")

structured_llm = llm.with_structured_output(RAGResponse)

prompt = ChatPromptTemplate.from_template(
    """
Based on the context below, answer the question.

Context:
{context}

Question: {question}

Provide a structured response."""
)

def format_docs(docs):
    return "\n\n".join(
        f"[{doc.metadata.get('source', 'unknown')}]: {doc.page_content}"
        for doc in docs
    )

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | structured_llm
)
result = rag_chain.invoke("What is LangGraph?")
print(f"Answer: {result.answer}")
print(f"Confidence: {result.confidence}")
print(f"Sources: {result.sources_used}")

Three things to take from it. First, when the terminal stage is with_structured_output(...), there is no StrOutputParser — the structured model already returns a RAGResponse instance, and appending a string parser here would break it. Second, the only instruction in that prompt is "provide a structured response": the schema does the work the prose would otherwise have to do, which is exactly why structured output is easier to validate than free text. Third, sources_used: List[str] makes citations come back as data rather than as prose you would have to parse out of a paragraph.

A caution about that confidence field: it is the model's self-report, elicited by a field description, not a calibrated probability. Treat it as a routing hint (send low to a human, or trigger a re-retrieval) and never as a measurement. Real confidence comes from retrieval scores (Part 4) and from evaluation (Part 7).

The whole runnable file, for reference

Here is the demo end to end, so you can see where the knowledge base, the splitter and the retriever settings sit relative to the chain:

demo_basic_rag() and its knowledge base — the complete, runnable path
Python
# production-course-main-code/rag_pipeline.py
# (imports as listed earlier in this part)
load_dotenv()
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")

KNOWLEDGE_BASE = """# LangChain Framework
... 1,010 characters of Markdown: Core Components, LangGraph, Pricing, Getting Started ...
"""

llm = init_chat_model(model="gpt-4o-mini", temperature=0.2)


def create_kb():
    """Create a vector store from knowledge base."""
    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    doc = Document(
        page_content=KNOWLEDGE_BASE, metadata={"source": "langchain_knowledge_base.md"}
    )
    chunks = splitter.split_documents([doc])
    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings_model,
        persist_directory=tempfile.mkdtemp(),
    )
    return vector_store


def demo_basic_rag():
    vector_store = create_kb()
    retriever = vector_store.as_retriever(
        search_type="similarity", search_kwargs={"k": 2}
    )

    prompt = ChatPromptTemplate.from_template(
        """
Answer the question based only on the following context:

{context}

Question: {question}

Answer:


Make sure to answer in a concise manner, 
and if you don't know the answer, just say "I don't know."""
    )

    def format_docs(docs):
        return "\n\n".join([doc.page_content for doc in docs])

    rag_chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )

    questions = [
        "What is LangChain?",
        "Who created LangChain?",
        "What is LangGraph used for?",
    ]
    for q in questions:
        answer = rag_chain.invoke(q)
        print(f"Q: {q}")
        print(f"A: {answer}\n")

Three things to notice, all of which become chapters later:

  • metadata={"source": ...} is set at ingestion time, before splitting. This is the single line that makes citations possible at query time; split_documents() propagates it onto every chunk. Lose it here and no downstream code can invent it.
  • persist_directory=tempfile.mkdtemp() means this demo re-embeds the whole corpus on every run. Fine for a 1,010-character knowledge base; a real bill and a real latency wall at scale. Use a stable directory in anything you run twice — this exact anti-pattern recurs in almost every demo file in the repo.
  • k=2 in the basic demo, k=3 in the sources and structured demos, k=2 again in the fallback demo. These are demo values chosen for a tiny corpus, not defaults you should inherit.

The demo knowledge base is a small Markdown string about LangChain, and two facts inside it are load-bearing for the demos: LangChain was created by Harrison Chase in October 2022, and LangSmith has a free tier with paid plans starting at $39/month. Those are the ground truth the Q&A demos are checked against — the $39/month fact is specifically the in-corpus probe used by the fallback demo below.

The last function in the file is worth reading even though it is labelled an exercise, because it is the first time the chapter's ideas are packaged as something reusable rather than a script — an object that owns its store, its retriever and its chain, and exposes one method:

DocumentQA — the same chain wrapped in a class you could actually import
Python
# production-course-main-code/rag_pipeline.py — exercise_document_qa()
class DocumentQA:
    def __init__(self, document: str, source_name: str = "document"):
        # Split document
        splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
        doc = Document(page_content=document, metadata={"source": source_name})
        chunks = splitter.split_documents([doc])

        # Create vector store
        self.vectorstore = Chroma.from_documents(
            documents=chunks,
            embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
        )
        self.retriever = self.vectorstore.as_retriever(search_kwargs={"k": 3})

        # Create chain
        self.llm = init_chat_model(model="gpt-4o-mini", temperature=0.2)

        self.prompt = ChatPromptTemplate.from_template(
            """
Answer based on the context. Rate your confidence (high/medium/low).

Context: {context}
Question: {question}

Format: [Confidence: X] Answer"""
        )

        def format_docs(docs):
            return "\n".join(d.page_content for d in docs)

        self.chain = (
            {
                "context": self.retriever | format_docs,
                "question": RunnablePassthrough(),
            }
            | self.prompt
            | self.llm
            | StrOutputParser()
        )

    def ask(self, question: str) -> str:
        return self.chain.invoke(question)

Same four stages, same k=3, same 500/50 splitter. source_name is a constructor argument because the metadata has to be decided by the caller at ingestion time — invariant: you cannot cite what you did not label.

Two caveats before you copy it. First, in the repo this class is declared inside the exercise_document_qa() function, so despite the shape it is not importable as written; lifting it to module scope is your first edit.

Second, this Chroma.from_documents(...) call passes no persist_directory at all, so the store is in-memory and dies with the process. Every DocumentQA(...) construction re-embeds the whole document and bills you for it. Give it a stable directory keyed by a content hash of the document, and the second construction becomes free.

The anti-hallucination prompt pattern

The instructor states this pattern three times in about ninety seconds 00:03:47, 00:04:49, 00:05:51. That repetition is the signal. It is four ingredients:

  1. The restriction clause — "Answer based only on the following context."
  2. The fallback clause — "If the context doesn't contain the answer, say 'I don't have information about that'" (or "I don't know").
  3. {context} — from the retriever branch.
  4. {question} — from the passthrough branch.

That is the whole thing.

Why it is worth doing first, before anything else in the guide: the prompt is the only grounding lever that costs nothing to pull. Changing it needs no retraining, no re-indexing and no new infrastructure — you edit a string, redeploy, and the behaviour changes on the next request. Every other fix in this guide (semantic chunking, hybrid search, reranking, contextual retrieval) requires re-embedding a corpus, standing up a component, or both.

"This is the low-hanging fruit when it comes to prompt engineering." 00:05:51

And the corollary, which is where most practitioners actually go wrong: it is "the low-hanging fruit... Yet a lot of people forget about it or don't pay attention to this" 00:05:51. The failure is not that people write a bad restriction clause. It is that they write none, then reach for reranking and better embeddings to fix a problem that two sentences of prompt would have contained.

The quantum-computing worked example

This is the example that makes the point stick 00:04:49. Same model, same question, same (irrelevant or empty) context. The only delta is one sentence in the prompt.

Without the instructionWith the instruction
Question"What is quantum computing?""What is quantum computing?"
Response"Quantum computing uses qubits…" — and then it keeps going, inventing"I don't have enough information pertaining to this topic."
CorrectnessFluent, coherent, ungroundedCorrect

Look closely at what the failure output does. It opens with a true statement. Quantum computing does use qubits. The fabrication grows out of a correct seed, in the model's normal confident register, in well-formed prose.

Hallucination usually starts from a true seed. That is what makes it hard to catch by reading.

Two rules fall out of this, and they are the reason Parts 7 and 13 exist at all:

Refusal is a feature you must explicitly request. Models do not abstain by default — their training rewards being helpful and continuing. An honest "I don't know" is the desired production output, not a system failure, and you have to ask for it in writing.

Coherence is not correctness.

"Even though it may sound coherent, but it is hallucinating, which means it's just making stuff up." 00:04:49

This is the single most transferable line in the source material. It means you cannot QA a RAG system by reading the answers. Fluency is what the model is optimised for; it is uncorrelated with groundedness. Any QA process built on "the answers look good" is measuring the wrong variable, which is precisely why this guide puts observability and evals (Part 7) ahead of the things they are usually put after, and why the checklist in Part 13 turns them into asserted tests rather than dashboards someone might look at.

The pattern, executable

The prompt in demo_basic_rag() above already carries both clauses. The dedicated fallback demo makes the refusal explicit and — importantly — ships a test design you should copy:

Python
# production-course-main-code/rag_pipeline.py — demo_rag_with_fallback()
prompt = ChatPromptTemplate.from_template(
    """
Answer the question based ONLY on the following context.
If the answer is not in the context, respond with: "I don't have information about that in my knowledge base."

Context:
{context}

Question: {question}

Answer:"""
)

questions = [
    "What is the pricing for LangSmith?",  # In knowledge base
    "What is the stock price of OpenAI?",  # Not in knowledge base
    "How do I deploy LangChain to AWS?",   # Not in knowledge base
]

The rest of that demo is the canonical chain unchanged — a plain format_docs, a k=2 retriever, prompt | llm | StrOutputParser(). Only the prompt text differs from demo_basic_rag(), which is the point: the refusal behaviour is bought entirely with the second sentence of a template.

That question list is a 1-in / 2-out probe, and it is the executable version of the quantum-computing contrast. One question the corpus can answer (the $39/month LangSmith pricing), two it cannot. A passing run answers the first and refuses the other two. A system that answers all three is hallucinating, and you found out in three seconds without an eval framework.

Note that the two out-of-corpus questions are chosen with care. "What is the stock price of OpenAI?" is about an adjacent entity the corpus mentions, and "How do I deploy LangChain to AWS?" is about the corpus's own subject in a direction it never covers. Both are the kind of near-miss the retriever will happily return something for. A probe made of questions about marine biology would pass trivially and tell you nothing.

Sources, citations and the trust chain

Grounding that you cannot prove is just a claim. Sources are the proof.

"we want to make sure that we also add some sources" 00:06:52

A retrieved Document is two fields: page_content (the chunk text) and metadata (a dict, carrying source — doc.pdf in the slide's example). Metadata is not decoration; it is the citation channel, and it exists only if ingestion put it there.

format_docs versus format_docs_with_sources

The difference is four lines, and it is the difference between a system that can be audited and one that cannot.

Python
# production-course-main-code/rag_pipeline.py — the sources-free formatter
def format_docs(docs):
    return "\n\n".join([doc.page_content for doc in docs])
Python
# production-course-main-code/rag_pipeline.py — demo_rag_with_sources()
def format_docs_with_sources(docs):
    formatted = []
    for i, doc in enumerate(docs):
        source = doc.metadata.get("source", "unknown")
        formatted.append(f"[{i+1}] {source}:\n{doc.page_content}")
    return "\n\n".join(formatted)

Each chunk becomes [n] <source>:\n<page_content>, and the numbered chunks are joined into one string. That string is the {context} the LLM reads. Four lines, zero extra API calls, no new dependency — the instructor is right to call this the canonical way to do RAG with sources, and there is no defensible reason to ship the first version instead.

The prompt has to ask for them, too. The sources demo does both halves:

Python
# production-course-main-code/rag_pipeline.py — demo_rag_with_sources()
prompt = ChatPromptTemplate.from_template(
    """
Answer the question based on the context below. Include which sources you used.

Context:
{context}

Question: {question}

Answer (include sources):"""
)

rag_chain = (
    {
        "context": retriever | format_docs_with_sources,
        "question": RunnablePassthrough(),
    }
    | prompt
    | llm
    | StrOutputParser()
)

Note retriever = vectorstore.as_retriever(search_kwargs={"k": 3}) here — three chunks, so three numbered sources. And note that the ending "Answer (include sources):" is doing real work: the formatter makes citation possible, the prompt makes it happen. Ship one without the other and you get nothing.

Two production-flavoured variants of the same idea ship in the same repo, and both are worth stealing:

Python
# production-course-main-code/research_assistant.py — _format_docs_for_context()
def _format_docs_for_context(self, docs) -> str:
    """Format retrieved documents into a string for the prompt."""
    if not docs:
        return "No relevant documents found."

    formatted = []
    for i, doc in enumerate(docs):
        source = doc.metadata.get("source", "Unknown")
        formatted.append(f"[Source {i+1}: {source}]\n{doc.page_content}")
    return "\n\n---\n\n".join(formatted)

The upgrades over the demo version: an explicit empty-result branch (retrievers return [] more often than you expect — a narrow metadata filter or an aggressive score threshold will do it, and an empty string in {context} is an invitation to hallucinate), and a visible --- separator so chunk boundaries are unambiguous to the model. The structured demo uses a third shape, f"[{doc.metadata.get('source', 'unknown')}]: {doc.page_content}", paired with a sources_used: List[str] field so citations come back as data rather than prose.

Why sources actually matter

"The reason why sources matter is because users are going to be able to verify those answers." 00:07:53

The chain is: sources → citations → verifiability → trust. And the last link is the one worth thinking hard about. The user does the final verification. The system's job is not to be believed; it is to make checking cheap. Give someone a filename, a page and a heading and they will spot-check three answers, find them right, and start trusting the system. Give them a confident paragraph with no provenance and they must either trust it blindly or re-do the research themselves — and the second option is why the system existed.

Citations reframe "accuracy" as "auditability" — and auditability is a property you can actually engineer, whereas accuracy is a property you can only measure.

There is a second audience too, and it is the one people forget: the system itself. Provenance is what lets the pipeline log which documents drove which answer, filter by permission before generation, deduplicate near-identical chunks, and — critically — debug. The instructor phrases it as the system's internal monologue:

"So it knows: okay, I got this answer, but I got it from where?" 00:06:52

Note the granularity hint in the same breath — "based on this document, based on this header, based on this place". Source granularity can be finer than the file. A Markdown header splitter that writes the header path into metadata gives you section-level citations for free, which is a Part 3 technique with a Part 1 motivation.

The mental model: two phases, one wrapper, three budgets

Everything in the remaining fifteen parts modifies exactly one box in this picture. Each part will tell you which one.

Figure 2The master map: both pipeline phases, the Parts covering each step, and observability
Original ASCII
PHASE A — INDEX (offline, once per document, amortized)
  Load ──► Split ──► [Contextualize] ──► Embed ──► Store + Index
  Part 2    Part 3        Part 14        Part 4      Parts 4, 8, 10

PHASE B — QUERY (online, once per request, recurring)
  Gate ──► Cache ──► Embed query ──► Retrieve ──► Fuse/Filter ──► Compress/Rerank
  P12       P9,P11       P4          P4,P6,P16        P6              P6
     ──► Assemble prompt ──► Generate ──► Validate output ──► Return + citations
          P1, P5, P9          P5, P11          P11              P1, P6

        ┌──────────────────────────────────────────────────────────┐
        │  OBSERVABILITY  (traces · metrics · evals · JSON logs)    │
        │  wraps everything, changes nothing           Part 7       │
        └──────────────────────────────────────────────────────────┘

  Governed by three budgets, always in tension: QUALITY · LATENCY · MONEY

The two-phase split is the most useful single distinction in RAG engineering, because the two phases have completely different economics. Phase A runs once per document and its cost amortises across every future query. Phase B runs once per request, forever. That asymmetry is a design instruction: push work left. Anything you can pre-compute at index time — contextual summaries, extracted metadata, cached embeddings, a second keyword index — buys you a thinner read path in exchange for a one-time bill.

The observability wrapper is drawn outside the boxes on purpose. It observes everything and changes nothing. When Part 7 builds it, the failure path records and re-raises — a monitoring layer that swallows an exception, retries, or alters a response has stopped being monitoring and become undocumented business logic. Part 13 then turns what the wrapper can see into assertions, which is the step most teams skip: a dashboard nobody is paged by is decoration.

The three budgets are the reason there are no universal right answers in this guide. Every technique you will read about buys one and spends another. Reranking buys quality and spends latency and money. Caching buys latency and money and risks quality (staleness). Bigger embeddings buy quality and spend money, latency and storage. Whenever a part gives you a default, it is a default for a particular point in that triangle, and it says which.

Also note where the production request pipeline puts its hardening layers — cheap rejections first, expensive inference last: Client → Rate limiter → Security (injection + PII) → Cache → Model (+retry +fallback) → Output validation → Metrics/logging → JSON response. Rejecting a hostile or duplicate request before it reaches an LLM is the difference between an incident and a log line.

The six invariants

These hold everywhere in this guide. When a later part appears to contradict one, it is making an explicit, argued exception — and it will say so.

1. Quality is inherited forward and never recovered downstream. Chunking → embeddings → retrieval → answer. You can have the best embedding model, the fastest vector database and the smartest LLM, and still get garbage if the chunking is wrong. Debug the earliest broken stage, not the visible symptom. The visible symptom is almost always the answer; the broken stage is almost never the generator.

2. One embedding model, both sides of the pipe, forever. The same model must embed your documents and your queries — and the version matters. Changing it invalidates the entire index. Mismatched dimensions fail loudly, which is the good case. Same-dimension-different-model fails silently, because the two models do not share a coordinate system: your search still returns k results, ranked confidently, and meaningless. Note the asymmetry: the generation model is swappable at will; the embedding model is a schema decision.

3. Index-time cost is one-time; query-time cost recurs per request. Push work left. Keep the read path thin.

4. Failures are silent. Nothing throws. Your error rate can be a clean 0% while your correctness is terrible. Retrieval returning the wrong chunks is a successful HTTP 200. This is why observability precedes optimization in this guide rather than following it.

5. Every layer is a "pick 2". The index picks two of accuracy / speed / memory. The infrastructure picks two of money / ops burden / control. The technique picks two of quality / cost / complexity. If a design appears to win all three, you have not found the axis it is spending on yet.

6. Monitoring observes and must never change behaviour. The failure path records and re-raises.

What Part 1 does not settle

Three things this part deliberately leaves open, so you do not mistake them for answered:

  • The five failure modes are a teaching taxonomy, not a partition. Real incidents combine them — bad chunking causes retrieval noise, which causes context overflow. Use the taxonomy to name the earliest cause, not to file the ticket.
  • The improvement figures quoted later are directional, not contractual. Contextual retrieval's 67% fewer retrieval failures and late chunking's 10–12% accuracy boost come from the part-6 README, which cites no methodology and no corpus. They are hypotheses to test on your own data, and Part 14 treats them as such.
  • The source course places observability after scaling. This guide does not. Every scaling and cost trigger in Parts 8–9 presumes instrumentation that can see a p95 latency or a drifting recall. Teaching them in the original order leaves you with triggers you cannot measure. That is a deliberate reordering, flagged here so you know it is ours.

Decisions to make before Part 2

Work through these now. Every one of them is expensive to change after you have embedded a corpus.

  1. Pick your embedding model — and write down its exact version string. It is a schema decision, not a config value (invariant 2). Declare it once as a module-level constant; never inline a model string at a call site. Part 4 gives you the dimension/price table to choose from.
  2. Design your metadata schema before you ingest anything. At minimum source. Strongly consider page, section/header, doc_type, created_at, and a tenant or permission key. You cannot filter or cite on a field you did not store, and the fix is a full re-index.
  3. Decide whether your corpus is structured or flowing. Headings and clear sections point at recursive chunking; unstructured flowing text where topics shift without markers points at semantic. Part 3 shows a measured A/B where the "better" technique loses 1–4.
  4. Build an evaluation corpus at production scale, not demo scale. If k is a meaningful fraction of your test corpus, your retrieval is untested.
  5. Write the out-of-corpus probe today. Three questions your corpus provably cannot answer, plus the exact refusal string, asserted in a test. Ship it before the first real query.
  6. Adopt the prompt pattern as a shared constant. Restriction clause + fallback clause + {context} + {question}. One definition, imported everywhere.
  7. Choose format_docs_with_sources over format_docs from the first commit. Retrofitting citations means retrofitting metadata, which means re-indexing.
  8. Agree the ordering rule with whoever owns the roadmap: build → debug → optimize → scale. Get it written down before someone asks you to "just scale it" on top of a retriever nobody has measured.

Part 1 checklist

Every line here is checkable against something that exists — a file, a test, a grep — not against a feeling that you understood the section.

  1. Both repos clone and import: uv run python -c "import langchain_chroma" succeeds in each of the two virtual environments.
  2. git grep -n "langchain_community import Chroma" over your own code returns nothing.
  3. You can write the canonical chain from memory — {"context": retriever | format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() — and say why the dict literal is a RunnableParallel.
  4. Nothing in your question branch modifies the string. Query rewriting, if you do it, lives in the retrieval branch only.
  5. Your prompt template is a named constant in one module, carries a restriction clause and a fallback clause, and is imported by every service that talks to the corpus.
  6. The fallback clause names an exact refusal sentence, and that same literal string appears in a test file.
  7. A test exists with at least three questions your corpus provably cannot answer, asserting that refusal string, and it runs in CI.
  8. Your ingestion sets metadata["source"] on the Document before split_documents(), and you have printed one retrieved chunk to confirm the field survived the split.
  9. Your formatter emits [n] <source>: markers inside the context string and your prompt asks the model to cite them. Both halves, or neither works.
  10. For the last change you made, you can name which box of the two-phase diagram it lives in and which of the three budgets it spent.
  11. You can name all five failure modes and the part of this guide that fixes each, without looking.
  12. You have stopped treating a 0% error rate as evidence of anything.

The instructor's own framing of what all this is for:

"By the end, you will have a production ready [RAG] that actually works." 00:01:44

Note the phrasing — that actually works — is a direct callback to the systems you have already built that didn't. That is the contract this guide inherits.

Next: Part 2 sets up the workshop — uv, provider keys, .env discipline — and gets real documents into a uniform list[Document] carrying the metadata that becomes these citations. As the source puts it at the end of the same breath: "All right, let's build."