Your First Real RAG System, and Debugging It When It Lies
Wire the first end-to-end chain with LCEL, then debug it systematically when it answers wrong.
Parts 2 through 4 built the pieces separately: a loader that turns files into Document objects, a splitter that cuts them into chunks, an embedding model that turns chunks into vectors, a Chroma collection that stores and searches them. Every one of those was a standalone demo function. None of them answered a question.
This part does two things, in the order the source course does them, because the order is the lesson.
First it assembles — chapter 9 [01:24:32–01:33:16] wires all five stages into one runnable LCEL chain that takes a string and returns an answer. The instructor is blunt that nothing in it is new; it is the same components as before, now in one place 01:30:38. RAG, at this level, is an assembly problem, not an algorithm problem. You will have a working system in about forty lines.
Then it destroys the illusion — chapter 10 [01:33:16–01:53:46] is the cold-shower chapter. The demo you just built answers three questions correctly over one Markdown string, and that proves essentially nothing. The chapter names the five ways tutorial RAG dies at scale, and more importantly it teaches the method: how to work backwards from a wrong answer to the stage that produced it, when no stage threw an exception.
That second half is the part most engineers skip and then need at 2am. Invariant 4 from Part 1 is doing all the work here: failures are silent. Your error rate can be a clean 0% while your correctness is terrible. The debugger is not a stack trace; it is a discipline.
The two-function architecture
Open rag_pipeline.py. Its shape maps one-to-one onto the two phases of the mental model from Part 1:
| Function | Phase | Runs | Cost profile |
|---|---|---|---|
create_kb() | Phase A — Index (offline) | Once per corpus | One-time, amortized |
demo_basic_rag() | Phase B — Query (online) | Once per question | Recurring, per request |
That is not a stylistic choice. It is Invariant 3 — index-time cost is one-time, query-time cost recurs — expressed as a Python function boundary. Any work you can move from the second function into the first, you should.
The module preamble sets up the two things that must be global:
# rag_pipeline.py:6-22 — the module preamble, verbatim (nothing removed)
from langchain_openai.embeddings import OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.chat_models import init_chat_model
from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from pydantic import BaseModel, Field
from typing import List
from dotenv import load_dotenv
import tempfile
load_dotenv()
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
Two things to notice before we go further, because both are traps.
load_dotenv() runs at line 21, before OpenAIEmbeddings(...) is constructed at line 22. The SDK reads OPENAI_API_KEY out of the process environment at construction time; reverse those two lines and you get an authentication error that looks like a bad key.
And embeddings_model is a module-level constant, declared exactly once. This is Invariant 2 made physical: one embedding model, both sides of the pipe, forever. Inline the model string at a call site and you have created a future silent-failure bug — two models of the same 1536 dimensions produce vectors that compare without erroring and rank meaninglessly.
The knowledge base is a string, not a file
The corpus is a Python string constant, KNOWLEDGE_BASE (lines 25–57): a 1,010-character Markdown document about LangChain, which the 500/50 splitter turns into exactly three chunks of 488, 414 and 123 characters. Its contents matter, because every answer the demo produces has to come from here and nowhere else, which is what makes it a usable test fixture — and its size matters just as much, because a three-chunk corpus cannot exhibit a single one of the failures the next chapter is about:
| Fact in the KB | Used as |
|---|---|
| LangChain created by Harrison Chase, October 2022 | Answer to test question 2 |
| 5 core components: Models, Prompts, Chains, Agents, Memory | Answer to the sources demo |
| LangGraph: state management, cycles and loops, human-in-the-loop, persistence | Answer to test question 3 |
| LangSmith free tier, paid plans from $39/month | The in-KB probe for the fallback demo |
pip install langchain langchain-openai; "first chain in under 10 lines of code" | Filler that must not be retrieved |
Two facts about that split are worth pausing on, because both are diagnostics you will reuse on your own corpus.
First, the arithmetic: 488 + 414 + 123 = 1,025 against a 1,010-character source. Chunks that sum to more than the input mean overlap actually fired. Here it fired once — chunk 2 ends on the heading ## Getting Started and chunk 3 opens by repeating it, 18 duplicated characters against the separators the splitter consumed, for a net surplus of 15. Chunks that sum to less than the input mean every cut landed on a separator and overlap bought you nothing.
Second, and less comfortable, the corpus is not the clean fixture it looks like.
create_kb() — the entire index path, in eighteen lines
# rag_pipeline.py:61-78
def create_kb():
"""Create a vector store from knowledge base."""
# split the knowledge base into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
doc = Document(
page_content=KNOWLEDGE_BASE, metadata={"source": "langchain_knowledge_base.md"}
)
chunks = splitter.split_documents([doc])
# create a vector store from the chunks
vector_store = Chroma.from_documents(
documents=chunks,
embedding=embeddings_model,
persist_directory=tempfile.mkdtemp(),
)
return vector_store
Four decisions are packed into that, and three of them are traps.
1. chunk_size=500, chunk_overlap=50. Recursive splitting at 500 characters with 50 of overlap — 10% — reused from Part 3 without re-justification. This is the "you have no better information yet" default. It is not a recommendation for your corpus; Part 3 measured recursive beating semantic 4–1 on heading-structured text, and chapters 10 and 14 are about tuning these two numbers with evidence.
2. The hand-built Document. There is no file to load, so the instructor constructs a Document himself with page_content= the raw string and metadata= a dict. His aside is the important part:
"because I'm creating the document myself, I can pass metadata as we saw before" 01:25:36
A Document is just a dataclass. When a loader will not give you the metadata your citations need, build the object yourself and attach whatever you want. The source value here — langchain_knowledge_base.md — is fictional; he picks .md because the string is Markdown, so downstream source attribution behaves the way it would with a real file.
3. split_documents([doc]), never split_text(...). Called out explicitly on camera at 01:25:36, and it is the whole reason step 2 exists.
split_text(str)returns a list of bare strings. Metadata is destroyed.split_documents([Document, ...])returns a list of Documents and **copies the parent's
metadata onto every chunk**.
Wrap the string in a Document so that split_documents has something to propagate. And note the brackets: split_documents takes a list, even for one document. Passing the bare Document is one of the most common errors in this API.
4. Chroma.from_documents(...) does three things in one call — it embeds every chunk with embeddings_model, writes vectors + text + metadata into a collection, and hands back a live store handle. Only documents and embedding are required.
The query path
The retriever is a thin adapter
# rag_pipeline.py:83-86
vector_store = create_kb()
retriever = vector_store.as_retriever(
search_type="similarity", search_kwargs={"k": 2}
)
A vector store is not a chain component. .as_retriever() adapts it into a Runnable whose contract is str → List[Document] — the only thing an LCEL pipe can consume. Two knobs are used here and both are worth arguing with:
search_type="similarity"— plain top-k nearest neighbour. The alternatives exist and are
not exercised: "mmr" (with search_kwargs={"k": 3, "fetch_k": 5} in vector_stores.py, fetch five and return three diverse ones) and "similarity_score_threshold", which is the only one that can return fewer than k when nothing is good enough.
search_kwargs={"k": 2}— two chunks per query 01:26:36. (The auto-captions render
search_kwargs as "the quirs", which is a useful reminder that the repo code, not the transcript, is authoritative for every name in this guide.)
search_kwargs is also where a metadata filter goes — {"k": 4, "filter": {"topic": "database"}} is the retriever-shaped form of the similarity_search(query, k=5, filter=...) call from Part 4. No demo in this file uses one, which is worth knowing: the retriever you are about to build has no topical guardrail whatsoever.
init_chat_model — and the import path that is easy to get wrong
# rag_pipeline.py:58 (module level, immediately after KNOWLEDGE_BASE)
llm = init_chat_model(model="gpt-4o-mini", temperature=0.2)
On camera this line is typed inside demo_basic_rag(); in the committed file it sits at module scope, which is where you want it — one client, shared by all four demos, constructed once per process.
The instructor starts typing ChatOpenAI, stops and switches 01:26:36, then visibly hunts for the import — trying langchain_core.chat_models, muttering "chat loaders" — before landing on it. The correct line is:
from langchain.chat_models import init_chat_model
It is not in langchain_core. init_chat_model is a provider-agnostic factory: give it a model string, get back the right chat model object, no provider-specific class import. His stated justification is just ergonomics — it is quicker to type 01:27:37 — but the real payoff is Invariant 2's asymmetry: the generation model is swappable, so it deserves a swappable constructor. The embedding model is not, so it gets a hard-coded module constant.
temperature=0.2, not the default, and chosen explicitly for determinism 01:27:37. Note it is 0.2, not 0 — a sliver of variability is left in. The reasoning is diagnostic, not aesthetic: if the model's temperature is high, two runs of the same question produce different answers, and you can no longer tell whether a changed answer came from your retrieval change or from sampling noise. Temperature near zero is a prerequisite for debugging retrieval at all. (chains_v1.py goes further and uses temperature=0 for its non-RAG demos.)
The prompt: three deliberate moves
# rag_pipeline.py:88-102
# RAG Prompt Template
prompt = ChatPromptTemplate.from_template(
"""
Answer the question based only on the following context:
{context}
Question: {question}
Answer:
Make sure to answer in a concise manner,
and if you don't know the answer, just say "I don't know."""
)
ChatPromptTemplate.from_template() builds a chat prompt from one template string with {} input variables. The two variables — {context} and {question} — are filled at runtime from the keys of the parallel dict that precedes the prompt in the chain. The instructor names the mechanism explicitly at 01:31:38: replacing those variables at runtime is the act of grounding the model.
Grounding = injecting retrieved evidence into the prompt at runtime so the model answers from your data rather than from its weights. It is a prompt-time operation. No retraining, no fine-tuning, ever.
Three design moves, each doing a specific job:
- "based only on the following context" — the restriction clause. It narrows the legal
answer surface to retrieved text.
- "answer in a concise manner" — output-length control. Cheap, and it keeps the response
inside a predictable token budget.
- "if you don't know the answer, just say 'I don't know.'" — the refusal escape hatch.
The third is the one people leave out, and leaving it out is a bug. "Answer only from context" tells the model what it may not do; it does not tell the model what to do instead. Without a sanctioned alternative output, an LLM asked to answer will answer. Refusal is a behaviour you must explicitly request.
format_docs — the glue at the type boundary
# rag_pipeline.py:104-106
# Format retrieved docs
def format_docs(docs):
return "\n\n".join([doc.page_content for doc in docs])
The retriever emits List[Document]. The prompt template needs a str. format_docs is the adapter, joining every chunk's text with a blank line between them so the chunk boundaries stay visible to the model 01:31:38.
The general lesson is bigger than this one function: every LCEL chain needs glue at its type boundaries, and LangChain makes that painless — a bare Python function placed in a | pipeline is automatically coerced into a RunnableLambda. You never write the wrapper. Nothing in the code says so, which is why it needs saying.
That coercion is one instance of the rule that makes the whole file readable. Every component here — retriever, prompt template, chat model, output parser, your own function — implements the same Runnable interface: .invoke(), plus .batch() and .stream() for free. Because they share one interface, | can compose any two of them, which is all LangChain Expression Language is. The practical consequence for debugging: any position in the chain can be invoked on its own, with the same .invoke() you use on the whole thing. That is what makes the bisection later in this part possible at all.
The heart: a dict literal is a parallel step
# rag_pipeline.py:108-114
# Rag chain
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
"And then here is the heart. This is where we create the actual rag chain." 01:31:38
A plain Python dict inside an LCEL chain is implicitly coerced into a RunnableParallel. The same input — the user's raw question string — is fanned out to every value in the dict, each branch runs concurrently, and the outputs are collected into a dict whose keys line up with the prompt's input variables. Two keys, two branches:
"context":question → retriever → List[Document] → format_docs → str"question":question → RunnablePassthrough() → the same question, untouched
RunnablePassthrough() is the identity Runnable, and it is load-bearing rather than decorative. The parallel dict consumes the single input; without an explicit identity branch, the original question is destroyed by the retrieval branch and {question} renders empty (or the template raises a missing-variable error). The instructor's phrasing at 01:29:38 is exactly right — its job is to make sure the incoming question is not changed.
There is a second, subtler reason it must be a passthrough and not, say, a rewriter: the user's exact wording is the thing being answered. Paraphrase it silently and you have changed the question without telling anyone.
StrOutputParser() terminates the chain by unwrapping the model's message object into a plain string, so rag_chain.invoke(q) yields something printable 01:30:38.
The full data flow, as narrated at 01:31:38–01:32:39:
Original ASCII
question (str)
→ {context: retriever | format_docs, question: RunnablePassthrough()} # parallel dict
→ prompt (ChatPromptTemplate fills {context} and {question})
→ llm (gpt-4o-mini, temperature=0.2)
→ StrOutputParser()
→ answer (str)Order in the pipe is also not negotiable. retriever | format_docs retrieves and then stringifies; reverse them and format_docs receives a string it will try to iterate. Type mismatches at a | boundary are the ordinary LCEL error.
Running it
# rag_pipeline.py:116-128
# Test the RAG chain
# Test
questions = [
"What is LangChain?",
"Who created LangChain?",
"What is LangGraph used for?",
]
print("Basic RAG Demo:\n")
for q in questions:
answer = rag_chain.invoke(q)
print(f"Q: {q}")
print(f"A: {answer}\n")
Three questions, a plain loop, .invoke() per question. The chain object is reusable and stateless across invocations — no memory, no chat history, each question completely independent. At 01:30:38 all three answer correctly.
That is the entire contract of basic RAG: str → str, with retrieval hidden inside. It is also the ceiling of basic RAG — no memory, no citations, no confidence, no refusal beyond what the prompt asks for.
Full listing — demo_basic_rag() (rag_pipeline.py:81-128)
def demo_basic_rag():
vector_store = create_kb()
retriever = vector_store.as_retriever(
search_type="similarity", search_kwargs={"k": 2}
)
# RAG Prompt Template
prompt = ChatPromptTemplate.from_template(
"""
Answer the question based only on the following context:
{context}
Question: {question}
Answer:
Make sure to answer in a concise manner,
and if you don't know the answer, just say "I don't know."""
)
# Format retrieved docs
def format_docs(docs):
return "\n\n".join([doc.page_content for doc in docs])
# Rag chain
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
# Test the RAG chain
# Test
questions = [
"What is LangChain?",
"Who created LangChain?",
"What is LangGraph used for?",
]
print("Basic RAG Demo:\n")
for q in questions:
answer = rag_chain.invoke(q)
print(f"Q: {q}")
print(f"A: {answer}\n")
The six-step review — read your own chain back in this order
When the instructor reviews the finished chain at 01:31:38–01:32:39 he walks it in a fixed order, and it is a good habit because it follows the data rather than the source file:
- Prompt first.
from_template, two input variables, replaced at runtime → grounding. - The formatter. How do retrieved documents become one clean string?
- The chain's dict.
contextandquestion— the heart. Which branch consumes the raw
input, and which one preserves it?
- The prompt template receives the dict. Do the dict keys match the template variables
exactly? A mismatch here is the most common runtime error in LCEL.
- The LLM receives the rendered prompt. Which model, which temperature?
- The parser turns the response into the final type.
StrOutputParserfor a string, or
with_structured_output for an object — never both.
If you cannot narrate those six steps for your own chain without scrolling, the chain is doing something you have not accounted for.
Environment, from pyproject.toml in the same repo: requires-python >=3.12 (and a .python-version pin of 3.12), langchain>=1.2.7, langchain-core>=1.2.7, langchain-openai>=1.1.7, langchain-chroma>=1.1.0, langchain-community>=0.4.1, python-dotenv>=1.2.1. Note the vector-store package: langchain-chroma, not langchain-community. The course's published README — which belongs to the second repo, fcc-production-rag-part-6, pinned at >=3.13 — says "Python 3.10+". Trust the pins, not the prose.
Three upgrades that ship in the same file
Chapter 9's video stops at demo_basic_rag(), but rag_pipeline.py contains three more chains that are strictly better starting points for anything real. None of them is demonstrated on camera. All three are worth adopting on day one.
1. Citations — sources travel inside the context string
# rag_pipeline.py:148-163 (demo_rag_with_sources, k=3)
def format_docs_with_sources(docs):
formatted = []
for i, doc in enumerate(docs):
source = doc.metadata.get("source", "unknown")
formatted.append(f"[{i+1}] {source}:\n{doc.page_content}")
return "\n\n".join(formatted)
rag_chain = (
{
"context": retriever | format_docs_with_sources,
"question": RunnablePassthrough(),
}
| prompt
| llm
| StrOutputParser()
)
Each chunk becomes [n] <source>:\n<page_content>, the prompt asks the model to "Include which sources you used", and the demo asks one question — "What are the core components of LangChain?", whose answer is the five-component list in the KB. The mechanism is the whole point: the source labels are part of the text the LLM reads. They are not passed out of band, not attached as metadata the model never sees. If the model cannot see the label it cannot cite it.
.metadata.get("source", "unknown") rather than ["source"] is deliberate defensive coding — a chunk whose loader did not set source degrades to the string unknown instead of raising KeyError mid-request.
2. Fallback — forcing a refusal string you can grep for
# rag_pipeline.py:176-210 (demo_rag_with_fallback, k=2)
prompt = ChatPromptTemplate.from_template(
"""
Answer the question based ONLY on the following context.
If the answer is not in the context, respond with: "I don't have information about that in my knowledge base."
Context:
{context}
Question: {question}
Answer:"""
)
# ...
questions = [
"What is the pricing for LangSmith?", # In knowledge base
"What is the stock price of OpenAI?", # Not in knowledge base
"How do I deploy LangChain to AWS?", # Not in knowledge base
]
The upgrade over "just say 'I don't know'" is that the refusal is a fixed, exact string. That turns abstention into something you can measure: count occurrences of that sentence in your logs and you have a refusal rate. A vague refusal is unmeasurable; a canonical one is a metric.
The three probe questions are a miniature eval set — one in-corpus (the $39/month LangSmith line), two out-of-corpus (OpenAI's stock price, AWS deployment). Build this set for your own corpus before you build anything else. It costs three lines and it is the only thing standing between you and "it seemed to work when I tried it."
3. Structured output — replacing the parser with a schema
# rag_pipeline.py:213-251 (demo_structured_rag, k=3)
def demo_structured_rag():
"""RAG with structured output."""
vectorstore = create_kb()
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
class RAGResponse(BaseModel):
"""Structured RAG response."""
answer: str = Field(description="The answer to the question")
confidence: str = Field(description="high, medium, or low")
sources_used: List[str] = Field(description="List of sources referenced")
follow_up: str = Field(description="Suggested follow-up question")
structured_llm = llm.with_structured_output(RAGResponse)
# ...
def format_docs(docs):
return "\n\n".join(
f"[{doc.metadata.get('source', 'unknown')}]: {doc.page_content}"
for doc in docs
)
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| structured_llm
)
Note what is missing: there is no StrOutputParser(). with_structured_output(RAGResponse) makes the model emit a validated Pydantic object, so the chain ends at the model and result.answer / result.confidence / result.sources_used / result.follow_up are attributes, not text you have to parse — which is exactly what the demo prints after invoking the chain on "What is LangGraph?". This is the shape you want the moment RAG sits behind an API (Part 11) rather than a terminal.
The prompt that goes with it is deliberately thin — "Based on the context below, answer the question… Provide a structured response." The field description= strings are carrying the instructions. That is the pattern: with structured output, the schema is part of the prompt, so put the wording where the fields are, not in prose the model has to reconcile with a schema.
confidence is self-reported and therefore not a calibrated probability — treat it as a weak signal for routing (e.g. "low → escalate to a human") and never as a correctness guarantee.
The reader exercise — DocumentQA, the whole pipeline as one class (rag_pipeline.py:271-312)
class DocumentQA:
def __init__(self, document: str, source_name: str = "document"):
# Split document
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
doc = Document(page_content=document, metadata={"source": source_name})
chunks = splitter.split_documents([doc])
# Create vector store
self.vectorstore = Chroma.from_documents(
documents=chunks,
embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
)
self.retriever = self.vectorstore.as_retriever(search_kwargs={"k": 3})
# Create chain
self.llm = init_chat_model(model="gpt-4o-mini", temperature=0.2)
self.prompt = ChatPromptTemplate.from_template(
"""
Answer based on the context. Rate your confidence (high/medium/low).
Context: {context}
Question: {question}
Format: [Confidence: X] Answer"""
)
def format_docs(docs):
return "\n".join(d.page_content for d in docs)
self.chain = (
{
"context": self.retriever | format_docs,
"question": RunnablePassthrough(),
}
| self.prompt
| self.llm
| StrOutputParser()
)
def ask(self, question: str) -> str:
return self.chain.invoke(question)
Tested on a four-fact blurb about Python (Guido van Rossum, first released 1991, Python 3.12 released October 2023, named after Monty Python) with three questions. Chunking 500/50, k=3, gpt-4o-mini at temperature 0.2, and a [Confidence: X] Answer output format.
Two flaws to fix before reusing it. The Chroma store is built with no persist_directory — it is in-memory, gone at process exit, and re-embedded on every construction. And format_docs here joins with "\n" rather than "\n\n", which visually merges adjacent chunks into what looks like one continuous passage; the double newline in demo_basic_rag exists precisely to keep chunk boundaries visible to the model.
The cold shower
Chapter 9 ends mid-sentence, on the setup line: you followed a RAG tutorial online and it all works 01:32:39. Chapter 10 completes the sentence at 01:33:18:
"the moment you add 10,000 documents, then everything breaks. Everything really everything breaks."
That same line is the epigraph of the course's second repo, fcc-production-rag-part-6: "You followed a RAG tutorial. It worked on 10 documents. Then you tried 10,000... and everything broke." Chapter 9's working demo is deliberately positioned as the naive baseline that the remaining twenty-three chapters tear apart.
Read that carefully, because the shape of the claim is the diagnostic insight. Nothing in the pipeline changed. The code is identical. The chunk size is identical. The embedding model is identical. The only thing that changed is the number of neighbours competing for the top-k slots.
This is why small-corpus success is not evidence. With ten documents there is almost nothing for a wrong answer to hide behind — retrieve two of ten and you have 20% of the corpus in context; retrieve two of ten thousand and you have 0.02%. Every chunking defect, every vocabulary mismatch, every ranking weakness is masked at small N and exposed at large N.
The chapter's organising claim is the course's headline: 90% of RAG systems fail in production, and they all fail for the same five reasons 01:33:18.
The five failure modes
| # | Failure | Symptom you observe | Broken stage | Fix | Owned by |
|---|---|---|---|---|---|
| 1 | Bad chunking | Answer is partial, or built from a fragment that reads like a sentence stub | Split | Semantic boundaries + overlap | Part 3 |
| 2 | Embedding mismatch (semantic drift) | The right chunk exists and is never retrieved; user vocabulary ≠ document vocabulary | Embed / Retrieve | Hybrid search, query rewriting, contextual retrieval | Parts 6, 14 |
| 3 | Retrieval noise | Right chunk is retrieved, buried among irrelevant neighbours; answer is diluted | Retrieve / Fuse | Reranking, filtering, compression | Part 6 |
| 4 | Context overflow | Everything was retrieved; the model ignores half of it and the answer truncates | Assemble prompt | Smart truncation, token budgeting, map-reduce | Part 9 |
| 5 | Hallucination | Answer contradicts a chunk that was correctly retrieved and correctly supplied | Generate | Constrained prompts + citations (here), evals that catch it (Part 7), output validation (Part 12), self-correcting retrieval (Part 15) | Parts 7, 12, 15 |
The instructor promises at 01:34:19 to dissect, diagnose and fix each one. In practice chapter 10 fixes #1, dissects #2 without fixing it, hands #3 to chapter 11, and leaves modes #4 and #5 for later. Worth knowing so you don't wait for a resolution inside this chapter that never arrives.
#1 — Bad chunking destroys meaning, silently
The chapter's working definition of a good chunk is semantic, not size-based:
A chunk is good if it still means something on its own after it has been torn out of the document.
The slide example is an HR remote-work policy cut in the wrong place:
- chunk A: "our policy allows up to"
- chunk B: "3 days per week individually"
Neither answers "how many remote days do I get?" Chunk A has the subject and no number; chunk B has the number and no subject. The meaning has been destroyed by the cut 01:35:20. The technical name is disjoint context.
The critical property is that the retrieval layer is working perfectly. It is faithfully finding and returning the nearest vector. The unit it is retrieving is broken. This is Invariant 1 in its rawest form: debug the earliest broken stage, not the visible symptom.
#2 — Embedding mismatch: the user says "cancel", the document says "termination policy"
"When user queries use different words than the words found in your documents, semantic search fails." 01:45:32
Both phrases mean the same thing to a human. To an embedding model they may sit far enough apart in vector space that the right chunk never enters the top-k at all — the query and the corpus are, in the instructor's word, not congruent 01:33:18.
What makes this one nasty is that it is invisible from the answer. The model gives you a confident, well-formed response built from the wrong chunks, and there is no signal anywhere that a better chunk existed and was passed over. Chapter 10 does not fix it — it takes you on a mechanics tour (single embedding → batch → cosine → ranking) so you can see numerically where the miss happens. The real remedies arrive later: BM25 catches the lexical miss (Part 6), query rewriting generates the vocabulary you didn't use (Part 6), contextual retrieval rewrites the chunk itself (Part 14, claimed +15–20% accuracy).
#3 — Retrieval noise: LLMs do get confused
The illustration at 01:33:18 is concrete and worth memorising as a ratio: the retriever is told to fetch 10 documents, dutifully returns 10, and only 2 of them are actually relevant.
The answer is in there. That is what makes this different from #2. The failure is dilution: eight irrelevant neighbours sitting alongside the two good ones.
"And so the LLM actually gets confused. Yes, LLMs do get confused." 01:52:41
That throwaway aside is the corrective to a very common and very wrong assumption — that a bigger context window makes retrieval quality irrelevant. Extra irrelevant chunks are not free padding; they are active distractors. Raising k to compensate for a weak retriever makes this worse, not better.
#4 — Context overflow
Too much material is stuffed into the prompt, the LLM ignores half of it, and answers get truncated 01:34:19. Note the double meaning the instructor uses and does not disambiguate: the context gets truncated (hard token limits) and the answers get truncated (the model runs out of budget or attention). This is the lost-in-the-middle problem stated informally.
#5 — Hallucination survives perfect retrieval
"Large language models are really good at hallucinating because they are good at ignoring context even though the answer is right there." 01:34:19
This is why hallucination is listed as its own failure mode rather than as a symptom of the other four. You can fix chunking, fix the vocabulary gap, eliminate the noise, stay inside the token budget, hand the model the exact right paragraph — and it can still invent an answer. Grounding reduces hallucination; it does not eliminate it. Any claim that RAG eliminates hallucination is simply wrong.
Why nothing throws
Here is the property that makes RAG debugging different from debugging ordinary software, and it deserves its own heading because every technique in the rest of this part exists to work around it.
All five failure modes produce a well-formed HTTP 200 response.
- Bad chunking: the embedding model faithfully encodes the fragment. The vector is not wrong —
it is an accurate representation of an incomplete fragment.
- Embedding mismatch: the retriever returns the k nearest vectors. There are always k nearest
vectors.
- Retrieval noise: the retriever returns exactly what you asked for.
kdocuments. - Context overflow: the prompt is valid, the model answers.
- Hallucination: the model produces fluent, plausible, grammatically perfect text.
Your error rate is 0%. Your latency is fine. Your dashboards are green. And your answers are wrong. This is Invariant 4, and it is why observability precedes optimization in this guide (Part 7) rather than following it.
Coherence is not correctness. You cannot QA a RAG system by reading the answer and deciding it sounds right. Every wrong answer this system produces will sound right — that is what the language model is for.
The debugging discipline
The method the chapter teaches, made explicit 01:33:16–01:53:46:
- Reproduce at scale, not on the toy corpus. A pipeline validated on ten documents tells
you almost nothing. If you cannot reproduce at 10,000, at least reproduce on a corpus large enough that the right chunk has real competition.
- Classify into one of the five buckets before touching code. Guessing which stage broke
and then editing it is how you spend a week tuning a splitter for an embedding problem.
- Inspect the retrieved chunks, not the answer. This is the whole trick, and it deserves
the next section.
- Fix the earliest broken stage. Chunking defects cannot be compensated for downstream.
Invariant 1 again.
Step 3 is the whole trick: bisect the pipeline
A RAG chain has exactly one seam you can cut cleanly: the boundary between retrieval and generation. Everything upstream of format_docs is retrieval. Everything downstream is generation. Cut there and you convert an unanswerable question ("why is this answer wrong?") into two answerable ones:
Was the right chunk retrieved?
- No → the bug is upstream. Stage: split (#1), embed/retrieve (#2). The LLM is innocent.
- Yes → the bug is downstream. Stage: assemble (#3, #4) or generate (#5).
This is Part 4's rule restated operationally — the course's claim is that 90% of RAG failures are retrieval failures, not generation failures — and the reason you must test retrieval separately before blaming the model. Swapping gpt-4o-mini for a frontier model to fix a retrieval bug is the most expensive no-op in this field: if you retrieve the wrong documents, no generation model can save you.
The retriever is a Runnable, so you can invoke it on its own. No chain, no LLM, no cost beyond one query embedding:
# Guide-authored diagnostic, built from the real APIs used in rag_pipeline.py
# and vector_stores.py. Run this BEFORE you touch the prompt.
def inspect_retrieval(vector_store, query, k=4):
"""Print exactly what the generation stage is about to be handed."""
retriever = vector_store.as_retriever(
search_type="similarity", search_kwargs={"k": k}
)
docs = retriever.invoke(query)
print(f"QUERY: {query}")
print(f"RETURNED: {len(docs)} of k={k}") # k is a ceiling, not a quota
for i, doc in enumerate(docs):
src = doc.metadata.get("source", "unknown")
print(f"\n[{i+1}] source={src} len={len(doc.page_content)} chars")
print(doc.page_content)
# Scores are a separate call: (Document, float) TUPLES, not Documents.
print("\n--- with scores ---")
for doc, score in vector_store.similarity_search_with_score(query, k=k):
# Chroma returns DISTANCE: 0 = identical, larger = worse.
similarity = 1 / (1 + score)
print(f" dist={score:.4f} sim={similarity:.4f} {doc.page_content[:70]}...")
What that one print harvests, mapped straight onto the taxonomy:
| What you see | Diagnosis |
|---|---|
| A chunk that reads like a sentence stub, or ends mid-thought | #1 bad chunking — go fix the splitter |
| The right chunk is absent entirely, and the query's words never appear in your corpus | #2 embedding mismatch |
| The right chunk is present at position 3 or 4, surrounded by junk | #3 retrieval noise — rerank or filter |
| The right chunk is present at position 1 and the answer still contradicts it | #5 hallucination — the LLM ignored it |
RETURNED is less than k | A filter or score threshold trimmed the list — k is a ceiling, not a quota, and downstream code assuming len(docs) == k will break |
Instrumenting the chain itself
Sometimes the seam you need is inside the chain. chains_v1.py ships three techniques, and they are the pure-LangChain half of what Part 7 later replaces with LangSmith:
# chains_v1.py:144-176
def demo_debbuging():
prompt = ChatPromptTemplate.from_template("Say hello to {name}")
chain = prompt | model | StrOutputParser()
# Method 1: Get configuration
print("Chain input schema:", chain.input_schema.model_json_schema())
print("Chain output schema:", chain.output_schema.model_json_schema())
# Method 2: Use with_config for tacing
result = chain.with_config(
run_name="greeting_chain",
# tags="demo,debugging",
).invoke({"name": "Alice"})
print(f"Greeting: {result}")
# Method 3: Inspect intermediate steps
# Using RunnableLambda for logging
def log_step(x, step_name=""):
print(f"[{step_name}] {type(x).__name__}: {str(x)[:100]}")
return x
debug_chain = (
prompt
| RunnableLambda(lambda x: log_step(x, "after_prompt"))
| model
| RunnableLambda(lambda x: log_step(x, "after_model"))
| StrOutputParser()
)
print("\nDebug chain execution:")
result = debug_chain.invoke({"name": "Debug"})
print(f"Greeting: {result}")
Method 1 — schemas. chain.input_schema.model_json_schema() and .output_schema tell you what the chain believes it accepts and returns. This is how you settle the raw-string-vs-dict argument from earlier without guessing.
Method 2 — with_config(run_name=...). Names the run so that when you do turn on tracing (Part 7) the span has a human label instead of a UUID. Note that tags="demo,debugging" is commented out for a reason: with_config(tags=...) expects a list[str], not a comma string. If you re-enable it, write tags=["demo", "debugging"].
Method 3 — RunnableLambda log taps. The important property, and the reason this pattern generalises: log_step returns x unchanged. It is a pure observer spliced into the pipe. That is Invariant 6 — monitoring observes and must never change behaviour — implemented in three lines. Drop one of these between retriever and format_docs and you get the retrieved documents printed on every request without altering the chain's semantics.
chains_v1.py also contains a fake-retriever RAG skeleton (demo_passthrough_chain) that is genuinely useful for isolation work — it runs the entire chain shape with a hard-coded context string and no vector store at all:
# chains_v1.py:86-100
# similuatee a retrieve operation
def fake_retriever(input_dict):
return " LangChain was created by Harrison Chase in 2022."
chain = (
RunnableParallel(
context=RunnableLambda(fake_retriever), question=RunnablePassthrough()
)
| RunnableLambda(
lambda x: {"context": x["context"], "question": x["question"]["question"]}
)
| prompt
| model
| StrOutputParser()
)
Substitute a stub retriever that returns the chunk you know is correct. If the answer is still wrong, retrieval was never the problem and you are looking at #4 or #5. If the answer is suddenly right, everything upstream is suspect. That is a clean bisection, and it costs one LLM call.
(The x["question"]["question"] double-unwrap at line 95 is an artefact of this file invoking with a dict — RunnablePassthrough() passes the whole input dict through. rag_pipeline.py's raw-string style avoids it, and itemgetter("question") is the cleaner idiom if you need the dict style.)
Stage 1 forensics: the splitter under the microscope
When the bisection says "the retrieved chunk is a fragment", you are debugging the splitter. text_splitters.py gives you the instruments.
Why recursive is the reliable default
Four linked properties, from 01:39:26–01:40:26:
- It preserves semantic coherence by splitting hierarchically through an ordered list of
separators, rather than cutting every N characters.
- The hierarchy encodes a theory of meaning. Paragraphs first, then sentences, then
words — because a paragraph should contain one thought, and so should a sentence 01:39:26. The splitter escalates to a finer separator only when the coarser one cannot satisfy chunk_size.
- It respects natural boundaries — it will not emit "unsupervised" and then cut before
"learning", because half a term is not a term.
- The payoff is downstream, not at index time. Chunking quality is never observable when
you chunk; it is observable weeks later, in answer quality. This is exactly why it is the hardest defect to attribute.
# text_splitters.py:89-100
def recursive_splitter():
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=["\n\n", "\n", " ", ""],
)
chunks = splitter.split_text(SAMPLE_TEXT)
print(f"Original length: {len(SAMPLE_TEXT)} chars")
print(f"Number of chunks: {len(chunks)}")
print(f"Chunk sizes: {[len(c) for c in chunks]}")
print(f"\nFirst chunk preview:\n{chunks[0][:200]}...")
SAMPLE_TEXT is a Markdown primer on machine learning — one # title, ## sections and two ### subsections, with bulleted algorithm lists — which is the point: it has real heading structure for the separators to find.
The separator list ["\n\n", "\n", " ", ""] is paragraphs → lines → words → characters. (The auto-captions render this as "double new lines, new lines also empty string and quotes", which is why the code is authoritative, not the transcript.) The empty string is the last resort: character-level cutting, which is fixed-size chunking by another name and produces the mid-word garbage Part 3 warns about.
split_text is correct here, and only here, because SAMPLE_TEXT is a raw string with no metadata to lose — the instructor says so as he types it 01:37:25. The rule from create_kb still stands: real pipelines hold Document objects and use split_documents.
Running it against the shipped file prints:
Original length: 887 chars
Number of chunks: 2
Chunk sizes: [462, 423]Read those three numbers as a diagnostic, not as trivia — it is the same arithmetic we ran on KNOWLEDGE_BASE, this time coming out the other way. 462 + 423 = 885 against an 887-char source: the cut landed on a \n\n, which is consumed, and nothing was duplicated — the splitter only carries a trailing fragment forward when that fragment fits inside the 50-character overlap budget, and whole paragraphs do not. So chunk_overlap=50 bought you nothing on this document. Conversely, when your chunk sizes sum to noticeably more than the source, the splitter is cutting inside paragraphs and you are paying overlap at every boundary. Two numbers, and you already know how the split went.
Overlap is cheap insurance
"So overlap essentially is what we call a cheap insurance." 01:43:31
The single strongest idea in chapter 10. Overlap means the trailing N characters of chunk i are repeated as the leading N characters of chunk i+1. Its purpose is to ensure that meaning straddling a boundary appears whole in at least one chunk, so that context at a boundary is never orphaned 01:45:32.
The failure it prevents is specific and maddening: a small, cheap redundancy is what stops your RAG system from holding the answer and being unable to find it 01:43:31.
The API-key worked example 01:44:32–01:45:32 — reproduce this one in your head every time you set chunk_overlap. Source text:
"The API key expires after 24 hours. You must refresh it using the token endpoint."
Without overlap:
| Chunk | Content | Contains |
|---|---|---|
| 1 | "The API key expires after 24 hours" | The problem, no solution |
| 2 | "You must refresh it using the token endpoint" | The solution, never says "expiration" |
The user asks: "How do I handle API expiration?" The retriever matches chunk 1 — it contains "expir…" — and returns it. The match is correct. The answer is useless. Chunk 2, which holds the actual fix, scores poorly because it never mentions expiration.
With overlap: chunk 2 also carries "…expires after 24 hours. You must refresh it", so both chunks contain problem and solution, and whichever one the retriever picks, the LLM gets a complete, actionable context.
"It is a difference between finding an answer and finding a complete answer. Those are two different things." 01:45:32
The demo that makes overlap visible:
# text_splitters.py:115-133
def overlap_importance():
text = "The quick brown fox jumps over the lazy dog. " * 10 # Repeated text
# without overlap
no_overlap = RecursiveCharacterTextSplitter(chunk_size=50, chunk_overlap=0)
# with overlap
with_overlap = RecursiveCharacterTextSplitter(chunk_size=50, chunk_overlap=20)
chunks_no_overlap = no_overlap.split_text(text)
chunks_with_overlap = with_overlap.split_text(text)
print("Without overlap:")
print(f" Chunk 1 end: ...{chunks_no_overlap[0][-20:]}")
print(f" Chunk 2 start: {chunks_no_overlap[1][:20]}...")
print("\nWith overlap:")
print(f" Chunk 1 end: ...{chunks_with_overlap[0][-20:]}")
print(f" Chunk 2 start: {chunks_with_overlap[1][:20]}...")
Two splitters differing in exactly one parameter — overlap 0 versus 20 — over a deliberately repetitive 450-character string, printing a 20-character window on either side of the first boundary. The actual output:
Without overlap:
Chunk 1 end: ...er the lazy dog. The
Chunk 2 start: quick brown fox jump...
With overlap:
Chunk 1 end: ...er the lazy dog. The
Chunk 2 start: the lazy dog. The qu...Without overlap the two chunks are strictly contiguous and share nothing — the phrase "the lazy dog. The quick brown fox" exists in the corpus and in no single chunk. With overlap, chunk 2 visibly repeats the tail of chunk 1, so a query about "lazy dog" now hits a chunk that contains the whole phrase instead of two chunks that each hold half of it.
That is the benefit. The demo also shows the bill, if you count: the same 450-character string yields 10 chunks at overlap 0 and 15 at overlap 20. Overlap is not free — it is a 50% increase in vectors to embed, store and search, on this document. "Cheap insurance" is a claim about the ratio, not about the cost being zero.
Sweeping the parameter you're about to guess at
# text_splitters.py:103-112
def chunk_size_comparison():
sizes = [200, 500, 1000]
print("=== Chunk Size Comparison ===")
for size in sizes:
splitter = RecursiveCharacterTextSplitter(
chunk_size=size, chunk_overlap=size // 5
) # 20% overlap
chunks = splitter.split_text(SAMPLE_TEXT)
print(f" Size {size}: {len(chunks)} chunks")
This function is not run on camera and is worth more than most that are. It sweeps 200 / 500 / 1000 and derives overlap as size // 5 — a 20% rule of thumb, at the upper end of Part 3's 10–20% band. Against the shipped 887-character SAMPLE_TEXT it prints:
=== Chunk Size Comparison ===
Size 200: 6 chunks
Size 500: 3 chunks
Size 1000: 1 chunksCompare that middle line with recursive_splitter(), which uses the same chunk size and the same document and produces 2 chunks, not 3. The only difference is overlap: 50 versus 500 // 5 = 100. Raising overlap manufactured an extra chunk. This is the quiet second-order effect of the parameter everyone tunes first — chunk count, embedding spend and index size are functions of both numbers, and at chunk_size=1000 the whole document collapses to a single chunk, at which point you have stopped doing retrieval altogether.
Run this against your own corpus, then run inspect_retrieval from earlier against each configuration and compare what actually comes back. That is the loop: sweep, retrieve, read the chunks. Chunk counts alone will not tell you which setting is better.
The rest of text_splitters.py — imported, shipped, and never run in this chapter
The file imports four splitter classes plus the Language enum, and this chapter demos exactly one of the splitters. Know what the rest are for before you need them (Part 3 owns the full argument):
MarkdownHeaderTextSplitter—markdown_splitter()(lines 136–149) splits on
[("#","h1"), ("##","h2"), ("###","h3")] and carries the header path into chunk.metadata. On SAMPLE_TEXT it yields 4 chunks whose metadata reads like {'h1': 'Introduction to Machine Learning', 'h2': 'Types of Machine Learning', 'h3': 'Supervised Learning'} — a breadcrumb trail per chunk. That metadata is what makes header-level citations possible. Never executed on camera.
Languageenum +from_language—code_splitter()(lines 152–160) uses
RecursiveCharacterTextSplitter.from_language(language=Language.PYTHON, chunk_size=500, chunk_overlap=50) against SAMPLE_CODE (797 chars: a quicksort and a binary_search), and cuts it into 2 chunks of 390 and 402 characters — one function each, which is the point. Chunking code with a generic splitter cuts functions in half and makes both halves useless as context.
document_splitter()(lines 163–180) — the only function in the file that uses
split_documents, over PyPDFLoader("./docs/langchain_demo.pdf"), printing the first and last chunk's metadata. This is the shape a real pipeline has.
CharacterTextSplitterandTokenTextSplitterare imported and never used anywhere in
the file.
document_splitter() is the one function left uncommented in __main__, so cloning the repo and running python text_splitters.py runs the PDF demo, not the two demos this chapter walks through. The PDF ships with the repo at docs/langchain_demo.pdf, but the path is relative, so the demo only runs from the repo root.
Stage 2 forensics: embeddings with the covers off
When the bisection says "the right chunk was never retrieved and the vocabulary differs", you are debugging the embedding stage. embeddings_deep.py is a deliberate de-abstraction: it rebuilds retrieval from numpy so you can watch the numbers.
Check the norm before you trust any score
Print np.linalg.norm(vector) on any embedding you get back.
- OpenAI embeddings are normalised to 1.0. On camera the norm prints as
1— the shipped
f"{...:.4f}" format string means you will see 1.0000.
- Without normalisation, raw magnitude leaks document length into similarity scores, and long documents win by being long rather than by being relevant.
- The diagnostic rule: norm == 1 → normalised (good); norm != 1 → not normalised. Not every model normalises, and the ones that don't will not tell you.
The instructor states the reason for normalisation compactly at 01:48:36:
"it will prevent longer documents from having bigger vectors just because they have more content."
This is a genuine portability gotcha, not a theoretical one. Swap OpenAIEmbeddings for HuggingFaceEmbeddings("sentence-transformers/all-MiniLM-L6-v2") (384 dimensions) or OllamaEmbeddings(model="llama2-7b-embedding-q4_0") — both referenced in embeddings.py — and you may silently move to an unnormalised space where your carefully-tuned thresholds mean nothing.
For text-embedding-3-small the demo prints 1536 dimensions and the first 5 vector components, so you can eyeball that you got floats and not an error object. (You have to repair the file first — see the bug callout below — because the two functions that print dimensions and norms are exactly the two that are broken as shipped.)
| Model | Dimensions | Cost / 1M tokens | Best for |
|---|---|---|---|
text-embedding-3-small | 1536 | $0.02 | General use — the guide's default |
text-embedding-3-large | 3072 | $0.13 | High accuracy (6.5× the price) |
text-embedding-ada-002 | 1536 | $0.10 | Legacy |
all-MiniLM-L6-v2 (HuggingFace) | 384 | local | Cheap/offline; check normalisation |
embed_query vs embed_documents
embed_query(text)→ one vector, for a query.embed_documents([t1, t2, t3])→ a list of vectors, for documents. The batch demo uses
3 texts.
The instructor's observation at 01:48:36 is that calling them feels identical — the only change is that embed_documents takes a list. The real reason to use the batch form is that the provider packs the whole list into one API round trip.
But there is a deeper reason LangChain keeps the two methods separate, and the chapter does not say it: some embedding models apply asymmetric prefixes or instructions to queries versus passages. Calling embed_documents on a query, or vice versa, can silently land you in a slightly different region of the space. Use the method that matches the role of the text.
The pre-RAG system
# embeddings_deep.py:35-65
def similarity_search():
# Documents
docs = [
"Python is a programming language",
"JavaScript is used for web development",
"Machine learning enables AI applications",
"Deep learning uses neural networks",
"Cats are popular pets",
]
query = "What programming languages exist?"
# embed documents and query
doc_vector = embeddings_model.embed_documents(docs)
query_vector = embeddings_model.embed_query(query)
# compute cosine similarities
def cosine_similarity(vec1, vec2):
return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
similarities = [cosine_similarity(query_vector, doc_vec) for doc_vec in doc_vector]
# rank documents by similarity
ranked_docs = sorted(zip(docs, similarities), key=lambda x: x[1], reverse=True)
print(f"Query: {query}\n")
print("Ranked by similarity:")
for doc, score in ranked_docs:
print(f" {score:.4f}: {doc}")
The formula is written by hand rather than imported, and that is the point:
cos(a, b) = dot(a, b) / (‖a‖ · ‖b‖)Five documents, one query, cosine against each, sort descending, print. That is a retriever with the covers off. Note the small irony in that denominator: if OpenAI's vectors really are unit-normalised, both norms are 1 and the division is a no-op — cosine similarity degenerates to a plain dot product. Writing it out anyway is correct defensive code, because the moment you swap in a model that does not normalise, the formula still holds and only the numbers move.
"So this is our pre-rag system essentially." 01:51:39 — "even though it's rudimentary … it mimics exactly what retrieval does and how it works."
The ranking is the evidence:
| Rank | Document | Reading |
|---|---|---|
| 1 (≈ 0.44) | "Python is a programming language" | Direct topical match |
| 2 | "JavaScript is used for web development" | Also a programming language — semantically near |
| 3–4 | "Machine learning enables AI applications" / "Deep learning uses neural networks" | Same broad domain, wrong topic |
| 5 | "Cats are popular pets" | Nothing to do with programming languages at all |
The top two are exactly what a human would pick, and neither the word "Python" nor "JavaScript" appears in the query. That is the semantic-search superpower in one output — and it is worth remembering, because Part 6 opens by showing the identical mechanism failing catastrophically on SKUs and error codes, where lexical exactness is the whole requirement.
Only the leading score, 0.44, is read aloud; the other four are not. Treat 0.44 as approximate — a re-run against text-embedding-3-small will differ in the third or fourth decimal.
The run also restates the symmetry rule 01:51:39: both the query and the documents must be converted into embeddings — through the same model, into the same space. That is Invariant 2, printed.
Stage 3 forensics: the right chunk is there and buried
If inspect_retrieval shows the correct chunk at rank 3 or 4, surrounded by plausible-looking noise, you have failure #3 and the splitter is innocent. Chapter 10 hands this to chapter 11, but advanced_rag.py in the same repo already contains the escalation ladder, and it is worth knowing what exists before you go build something.
advanced_rag.py even ships the right test fixture. INFO_BURIED is a two-document corpus where the answer is deliberately drowned: one paragraph about LangChain sits inside an ACME company-history document full of Series B funding, PTO policy, revenue figures and 401(k) matching, and one paragraph about LangGraph sits in the middle of a five-chapter platform runbook about EKS, Auth0 and DataDog. That is what retrieval noise looks like as data — build the same thing for your corpus.
| Technique | What it fixes | Real code | Cost |
|---|---|---|---|
| Multi-query | #2 vocabulary mismatch — generates several phrasings of the question | MultiQueryRetriever.from_llm(retriever=..., llm=...) over k=2 | +1 LLM call per query |
| Contextual compression | #3 noise, #4 overflow — extracts only the query-relevant sentences from each chunk | ContextualCompressionRetriever(base_compressor=LLMChainExtractor.from_llm(llm), base_retriever=...k=4) | 1 LLM call per retrieved doc |
| Hybrid / ensemble | #2 lexical misses — BM25 keyword + dense vector | EnsembleRetriever(retrievers=[bm25, semantic], weights=[0.4, 0.6]), both at k=3 | +20–50 ms |
| Parent-document | #1 fragments — search small chunks, return large ones | ParentDocumentRetriever(child_splitter=200/20, parent_splitter=800/100) | Extra docstore |
# advanced_rag.py:260-273 (demo_ensemble_hybrid_search) — hybrid search, the failure-#2/#3 workhorse
vectorstore = create_base_vectorstore()
# BM25 keyword retriever
bm25_retriever = BM25Retriever.from_documents(TECH_DOCS)
bm25_retriever.k = 3
# Semantic retriever
semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
# Ensemble combines both
ensemble_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, semantic_retriever],
weights=[0.4, 0.6], # 40% keyword, 60% semantic
)
Note the shipped weights are [0.4, 0.6] — 40% keyword, 60% semantic — while the course narration recommends starting at [0.5, 0.5]. Both are defensible; the point is that the weight is a dial you tune against a test set, and Part 6 owns that argument along with RRF fusion, k=4, and BM25's lack of incremental updates.
# advanced_rag.py:346-348 (demo_parent_document_retriever) — parent-document: small to search, large to answer
# Splitters
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
Small children (200/20) give precise vector matches; large parents (800/100) give the LLM a complete thought. It is a structural answer to the same disease overlap treats chemically.
The complete advanced chain stacks two of them — multi-query for recall, then compression for precision — and then reuses the exact same LCEL shape you built at the start of this part:
# advanced_rag.py:397-430 (demo_advanced_rag_chain, excerpt)
# Multi-query for better recall
multi_retriever = MultiQueryRetriever.from_llm(
retriever=vectorstore.as_retriever(search_kwargs={"k": 3}), llm=llm
)
# Compression to focus on relevant info
compressor = LLMChainExtractor.from_llm(llm)
advanced_retriever = ContextualCompressionRetriever(
base_compressor=compressor, base_retriever=multi_retriever
)
# ...
rag_chain = (
{"context": advanced_retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
That is the payoff of the LCEL shape. Every improvement in Parts 6 through 16 is a substitution at one position in this pipeline — a different retriever, a different formatter, a different parser. The chain you wrote in forty lines is the last piece of architecture you write.
Working the taxonomy backwards
Print this table. It is the index from symptom to stage.
| Symptom | First check | Likely mode | Go to |
|---|---|---|---|
| Answer is a fragment or trails off mid-thought | Read the retrieved chunk text | #1 chunking | Part 3 |
| Answer is confidently wrong; the right chunk is not in the retrieved set; query wording differs from the corpus | inspect_retrieval — is the target chunk absent? | #2 embedding mismatch | Parts 6, 14 |
| Right chunk retrieved at rank 3–4 among junk | Rank position of the target chunk | #3 retrieval noise | Part 6 |
| Long context; answer ignores the middle or truncates | Token count of the assembled prompt | #4 context overflow | Part 9 |
| Right chunk at rank 1; answer contradicts it | Diff the answer against docs[0].page_content | #5 hallucination | Parts 7, 12, 15 |
| Answer changes between identical runs | temperature | Not a RAG bug | Set 0–0.2 |
Fewer results than k | A filter or score threshold | Not a bug — k is a ceiling | Part 4 |
| Everything worked yesterday, nothing today, no code change | Did the embedding model or its version change? | Invariant 2 violation | Re-index |
And the rule that governs the order you work in:
Build → debug → optimize → then scale. Scaling a low-quality retriever produces wrong answers faster and more expensively. Every part after this one presumes you can already tell which stage broke.
Note on the source material
Two corrections this part applies to the source, stated plainly rather than smoothed over:
- **Slide 07 (
07-why-llm-debugging-is-hard.png) is indexed to chapter 10 and does not belong
here.** Its content — non-deterministic, cascading errors, silent failures, cost surprises — appears nowhere in chapter 10's transcript, and all four terms appear in chapter 12 (observability, 02:21:10). It is used in Part 7 of this guide. Chapter 10's actual on-screen slides — the five-failure-modes list, the "our policy allows up to / 3 days per week" illustration, and the API-key overlap example — were not captured as screenshots.
- The "smart truncation" announcement at 01:53:42 is misfiled as the fix for retrieval
noise; it is delivered at the end of chapter 11 as the fix for context overflow. See the callout above.
Chapter 10 ends mid-sentence, on the handoff line into hybrid search 01:53:42:
"We know that vector search is amazing."
That is where Part 6 begins — with the cases where it is not.
Checklist
Each item names the check, not just the goal. If you cannot run the check, the box is not ticked.
Before you call the system "working":
load_dotenv()appears above the first client constructor in the file. Read the import block top to bottom and confirm the order.- The embedding model is a module-level constant, declared once.
grep -c 'OpenAIEmbeddings(' yourfile.pyreturns 1. - You used
split_documents([doc])— with the list — notsplit_text. Then assertchunks[0].metadata["source"]exists; if it raises, you used the wrong method. - Every chunk carries a meaningful
source. Anything you may later want to filter or cite on is in the metadata at index time — it cannot be added afterwards without re-indexing. persist_directorypoints at a stable path, nottempfile.mkdtemp(). Run the indexer twice and confirm the second run makes no embedding API calls.- Print the chunk boundaries once, by hand.
[len(c.page_content) for c in chunks]plus the first and last 60 characters of each. If a numbered list, a table or a definition is cut in half, fix the splitter now — the demo corpus in this very chapter fails this check. temperatureis 0 or 0.2. Ask the same question twice and diff the answers; if they differ, you cannot attribute anything.- The prompt has all four ingredients: restriction clause, refusal clause,
{context},{question}. - The refusal string is exact and greppable, so
grep -cover your logs is a refusal rate. - You know whether your chain takes a raw string or a dict, and you checked with
chain.input_schema.model_json_schema()rather than by trying both. - You have three probe questions — one certainly in the corpus, two certainly outside it — and they live in the repo, not in your shell history.
Before you call a wrong answer a bug in the LLM:
- You ran the retriever alone and read the chunk text, not the answer.
- You printed
doc.metadata['source']next to every result. - You checked the polarity of your score — distance or similarity? — before applying any threshold.
- You checked
len(docs)againstk(a filter or score threshold may have trimmed it). - You checked
np.linalg.norm(vector)if you have changed embedding provider. - You spliced a
RunnableLambdalog tap betweenretrieverandformat_docsand looked at what the prompt was actually handed, rather than what you assume it was handed. - You substituted a stub retriever returning the chunk you know is correct. Still wrong → the bug is #4 or #5 and lives downstream. Suddenly right → everything upstream is suspect.
- You classified the failure into one of the five buckets before editing code, and you can say the bucket number out loud.
- You fixed the earliest broken stage, not the most visible one.
- You changed exactly one variable and re-ran the same probe questions.
Decisions to make now, that are expensive to change later:
| Decision | Default to start | Reversible? |
|---|---|---|
| Embedding model + version | text-embedding-3-small, 1536 dims, $0.02/1M | No — changing it invalidates the whole index |
| What goes in chunk metadata | source at minimum; add anything you may filter or cite on | No — requires re-indexing |
| Chunk size / overlap | 500 / 50 characters, then sweep 200/500/1000 at 20% overlap | Yes, but requires re-indexing |
k | 4, not 2; raise only after fixing rank quality | Yes, free |
| Persistence | Stable persist_directory from day one | Yes, but you re-embed |
| Generation model | gpt-4o-mini, temperature 0.2 | Yes — swap freely, that's the asymmetry |
| Output shape | StrOutputParser for the terminal; with_structured_output behind an API | Yes |