Agentic RAG: Self-Correcting Retrieval with LangGraph
Self-correcting retrieval: grade, rewrite and retry with a LangGraph state machine.
Every pipeline you have built so far in this guide is a straight line. Documents go in one end, an answer comes out the other, and the line runs exactly once per question. Part 3 improved what goes into the index. Part 6 improved what comes out of the retriever. Part 14 improved what each chunk means before it is embedded. All of those are still one-shot: you get one retrieval, and whatever it hands you is what the generator has to work with.
This part changes something different. It changes the control flow of Phase B. Instead of retrieve → generate, you get retrieve → grade → {generate | rewrite → retrieve | give up} — a graph with a cycle in it, where the system reads its own retrieval results, decides they are not good enough, and tries again with a better query.
That is the whole idea, and it directly attacks invariant 4 from the mental model: failures are silent. One-shot RAG cannot tell you it failed, because nothing in it ever looks at the retrieval output and asks "is this actually about the question?" Agentic RAG inserts exactly that question into the pipeline, and gives the system somewhere to go when the answer is no.
It is also the most expensive thing in this guide per query, and the chapter that teaches it never puts a number on the bill. This part does.
The problem: a result is not the same as a good result
The instructor frames traditional RAG at 06:39:44 — query, retrieve, pass to the LLM — with a sentence worth sitting with:
"After that you have a result whether it's a good result or not, it is a result."
That is the entire indictment. A one-shot RAG pipeline has no notion of success. If the retriever returns three documents about Python's release history when the user asked how to make pizza, the pipeline does not notice. It formats those three documents into a context block, hands them to the model with a polite instruction, and produces prose. Depending on how strict your prompt is, you get either a hallucination or a confused refusal — and in both cases your error rate is 0%, because nothing threw.
This is the same disease Part 1 named when it separated fluency from correctness, now with a mechanism attached. The pipeline is not broken; it is unsupervised. There is no stage whose job is to judge the previous stage.
Agentic RAG's contribution is one word inserted into the pipeline: evaluate. Everything else — the graph, the state object, the retry budget — is the plumbing needed to act on what the evaluation says.
The behavioural definition of self-correcting retrieval, in the instructor's words at 06:43:05:
"Okay, this didn't work. Let's go back and shuffle until we get to the correct result."
The formal version, at 07:01:26: the system grades what the retriever returned and checks whether it is "actually congruent with the query that came in" — and if not, increments the retry counter and rewrites.
The instructor's one-line contrast, at 06:43:05:
"Traditional RAG: query, retrieve and generate — so, one shot. With agentic RAG… we evaluate, and we decide if it needs to be re-retrieved."
What "agentic" actually means here
Before the graph, the vocabulary. The course defines an agent at 06:39:44 as something "smarter than just a large language model" — one that uses an LLM as its base but can call tools and, most importantly, make decisions.
Four patterns follow from that definition, and the chapter names all four:
| # | Pattern | What it means | In this chapter's code? |
|---|---|---|---|
| 1 | Routing | "If this is good, go this way. If it's not, go back this way." 06:40:44 | Yes — the conditional edge |
| 2 | Self-correction | Detect bad results, rewrite the query, retry 06:41:44 | Yes — the rewrite → retrieve cycle |
| 3 | Multi-step | Break a complex query into sub-queries 06:41:44 | No — asserted, never implemented |
| 4 | Tool use | Call tools and decide when 06:42:04 | No — one hard-wired retriever |
Pattern 4 is where the instructor puts the emphasis, and the emphasis is on the second half:
06:42:04 *"Agents are also able to go and call tools and decide when to call those tools."*
Deciding when is the load-bearing part. A chain that always calls the same tool in the same order is not an agent; it is a chain with extra syntax. Be honest with yourself about which one you have built.
The natural extension the chapter describes but does not code (06:42:04) is multiple retrieval tools, each implementing a different strategy — dense vector search, BM25, hybrid, parent-child, a different collection entirely. When tool 1 returns suboptimal results, the agent effectively says "okay, you don't work well because the results are suboptimal" and escalates to tool 2. That is the strategy-level sibling of query rewriting:
Rewrite the query, or change the retriever. Those are the two levers a self-correcting retrieval loop has. The lesson implements only the first; the retrievers for the second are already in your toolbox from Part 6.

Why LangGraph — the technical reason, not the marketing one
At 06:43:05 LangGraph is introduced as the part of LangChain "that allows you to easily create AI agents" and, more usefully, one that "lets you build these decision loops easily."
The real reason is narrower and worth stating precisely, because it is the only thing that forces a new framework into your stack:
LCEL chains are acyclic.
prompt | llm | parseris a directed pipe; there is no way to express "go back to step 2" in it. A retry loop is a cycle.StateGraphis the thing that adds cycles.
Everything else LangGraph gives you here — nodes, edges, a typed state object — you could approximate with a while loop and a dictionary. What you could not approximate cheaply is the surrounding machinery that comes free once your control flow is a graph object: checkpointing, per-node streaming, human-in-the-loop interrupts, and per-node tracing. None of those are used in this lesson, but they are why you build the loop as a graph instead of as a while loop.
The file's own docstring is blunt: "LangGraph 1.x is the production standard for building these workflows." Treat "production standard" as the instructor's opinion in April 2026, not a fact — he flags the dating himself at 07:02:28: "this is the production pattern you need to use in 2026 and moving forward. Things may change later."

04_agentic_rag.py ("Lesson 6.4: Agentic RAG with LangGraph"), Python 3.12.2 in the recorded run. The four bullets are the chapter's contract: evaluate relevance, reformulate and retry, use multiple retrieval strategies, self-correct and iterate. The shipped code delivers the first, second and fourth; "multiple retrieval strategies" is aspiration.Environment, imports and version pins — including a contradiction worth knowing about
The file is 04_agentic_rag.py in the Part 6 course repo (fcc-production-rag on screen, checked out locally as fcc-production-rag-part-6). Its requirements.txt is headed "Part 6: Advanced RAG - Dependencies (April 2026)"; the pins that matter here:
langchain>=1.2.0,<2.0.0
langchain-core>=1.3.0,<2.0.0
langchain-openai>=1.1.0,<2.0.0
langchain-chroma>=1.1.0,<2.0.0
chromadb>=1.5.0,<2.0.0
langgraph>=1.1.0,<2.0.0
langchain-classic>=1.0.0The import block of the lesson file, verbatim — note lines 21–22:
import os
from typing import TypedDict, Annotated, Literal
from operator import add
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.documents import Document
from langgraph.graph import StateGraph, END
load_dotenv()
os, Annotated and add are imported and never used, and Chroma is imported twice. Also worth noticing: this file takes TypedDict from typing, while every LangGraph file in the sibling repo (langgraph_core.py, conditional_edges.py, cycles_loops.py) takes it from typing_extensions. Two repos in the same course, two conventions; pick one deliberately.
The source contradicts itself on the Python version. pyproject.toml requires Python >= 3.13, .python-version says 3.13, and the VS Code status bar in the recorded run says Python 3.12.2. The recording predates the pin bump. Trust the pin, not the video. (The "requires Python 3.10 or higher" line you will see in the demo corpus is sample-document text written for the grader to score, not a repo pin — don't read it as one.)
Caption trap. The auto-generated captions in this chapter render agentic_rag_demo as "a gent demo", ChatOpenAI as "shadow openi", StateGraph as "site graph", and "grader" as "greater". Always check the code before quoting an identifier from this course.
The state object is the agent's memory
Everything in a LangGraph workflow flows through one shared dictionary. Nodes do not call each other and do not pass arguments; each node reads from state and returns a partial dict that LangGraph merges back into state.
06:44:06 "Think of a state as the memory" — "a lingering memory where all these nodes… are going to be able to write to and read from."
Later, at 06:47:09, the same idea again: the state is "the source of truth per se where all of the pieces are congruent." That is not decoration — it is why the router at the end of the graph needs no arguments. Everything it must know has already been written into state by the nodes that ran before it.
The schema is a plain TypedDict, and the file says why:
# fcc-production-rag-part-6/04_agentic_rag.py (lines 35-49)
class RAGState(TypedDict):
"""
State schema for our agentic RAG workflow.
LangGraph uses TypedDict for state (not Pydantic in 1.x).
The Annotated[list, add] tells LangGraph to merge lists.
"""
query: str
rewritten_query: str
documents: list[Document]
generation: str
relevance_score: float
retry_count: int
max_retries: int
Seven fields, each with a distinct job:
| Field | Written by | Read by | Why it exists |
|---|---|---|---|
query | caller (never overwritten) | grade, rewrite, generate, fallback | The user's actual intent, preserved through every retry |
rewritten_query | rewrite | retrieve | What retrieval actually uses once a rewrite has happened |
documents | retrieve, then overwritten by grade | grade, router, generate | The working set; graded down to survivors |
generation | generate or fallback | caller | The answer, real or honest-failure |
relevance_score | grade | router | Average of per-document grades |
retry_count | rewrite (increments) | router, fallback | How many times we have gone round |
max_retries | caller | router | The leash |
The last two are the ones that matter most in production, and the chapter hammers them:
Merge semantics, and the reducer the file documents but never uses
A node returns {"documents": [...]} and LangGraph merges that key into state, leaving the other six untouched. By default the merge is replace. If you want a field to accumulate across loop iterations instead — an audit trail of every attempt, for example — you annotate it with a reducer function. That is what Annotated[list, add] in the docstring refers to. The lesson file imports both Annotated and add and then never uses either; no field in RAGState carries a reducer, so every write is a replacement.
The working example is in the sibling repo:
# production-course-main-code/langgraph_core.py (lines 64-67)
class AccumulatingState(TypedDict):
messages: Annotated[list[str], operator.add] # lists concatenate when merged
count: Annotated[int, operator.add] # counts sum when merged
And in the self-correcting code generator, the same idiom is used for exactly the purpose an agentic RAG loop should copy — keeping every error from every attempt:
# production-course-main-code/cycles_loops.py (lines 19-26)
class CodeGenState(TypedDict):
task: str
code: str
errors: Annotated[list[str], operator.add]
iteration: int
max_iterations: int
success: bool
The five nodes
Every node has the same signature: f(state: RAGState) -> dict, and every one returns only the fields it changed. Two of the five (retrieve, fallback) make no LLM call at all; the other three each construct their own ChatOpenAI client inline — convenient for a lesson, wasteful in a server, where you would hoist those to module level or inject them.
| Node | LLM calls | Reads from state | Writes to state |
|---|---|---|---|
retrieve | 0 (1 embedding) | rewritten_query, query | documents |
grade | one per document | query, documents | documents (filtered), relevance_score |
rewrite | 1 | query, retry_count | rewritten_query, retry_count |
generate | 1 | query, documents | generation |
fallback | 0 | query, retry_count | generation |
1. retrieve — the only node that touches the vector store
# fcc-production-rag-part-6/04_agentic_rag.py (lines 119-142)
def retrieve_documents(state: RAGState) -> dict:
"""
Retrieve documents based on the query.
Uses rewritten_query if available, otherwise original query.
"""
query = state.get("rewritten_query") or state["query"]
print(f"\n[RETRIEVE] Searching for: '{query}'")
vectorstore = state.get("_vectorstore") # Injected at runtime
if not vectorstore:
# Fallback - create new (in production, pass via config)
vectorstore = create_sample_vectorstore()
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
documents = retriever.invoke(query)
print(f"[RETRIEVE] Found {len(documents)} documents")
for i, doc in enumerate(documents, 1):
print(
f" {i}. {doc.metadata.get('source', 'unknown')}: {doc.page_content[:50]}..."
)
return {"documents": documents}
Three things to notice.
The first line is the whole loop mechanism. state.get("rewritten_query") or state["query"] means: use the rewrite if one exists, otherwise the original. On the first pass rewritten_query is "" (falsy), so the original query is used. After a rewrite, retrieval silently switches to the new phrasing. One expression, and the node works identically on attempt 0 and attempt 3.
k=3 is hardcoded and it is the cost multiplier. The grader will make one LLM call per document returned here. k is not just a retrieval knob any more; it is a direct multiplier on your grading bill.
The logging is not decoration. Printing metadata["source"] plus the first 50 characters of each hit, on every attempt, is what makes the loop debuggable at all — and it is the same habit Part 4 told you to adopt. In production replace print with the structured JSON logger from Part 7, keyed on a request id, so you can reconstruct a five-node, three-attempt run from logs alone.
2. grade — the node that makes it agentic
This is the new stage. The file's own comment says so: "This is the KEY difference from traditional RAG - we evaluate before generating."
# fcc-production-rag-part-6/04_agentic_rag.py (lines 145-206)
def grade_documents(state: RAGState) -> dict:
"""
Grade retrieved documents for relevance to the query.
This is the KEY difference from traditional RAG - we evaluate before generating.
"""
query = state["query"]
documents = state["documents"]
print(f"\n[GRADE] Evaluating {len(documents)} documents for relevance...")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
grading_prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"""You are a relevance grader. Given a user query and a document,
determine if the document contains information relevant to answering the query.
Output ONLY a number between 0 and 1:
- 1.0 = Highly relevant, directly answers the query
- 0.7 = Somewhat relevant, contains related information
- 0.3 = Marginally relevant, tangentially related
- 0.0 = Not relevant at all
Output ONLY the number, nothing else.""",
),
(
"human",
"""Query: {query}
Document: {document}
Relevance score (0-1):""",
),
]
)
# Grade each document and calculate average
scores = []
relevant_docs = []
for doc in documents:
chain = grading_prompt | llm
result = chain.invoke({"query": query, "document": doc.page_content})
try:
score = float(result.content.strip())
except ValueError:
score = 0.5 # Default if parsing fails
scores.append(score)
print(f" - {doc.metadata.get('source', 'unknown')}: {score:.2f}")
if score >= 0.5: # Keep documents with score >= 0.5
relevant_docs.append(doc)
avg_score = sum(scores) / len(scores) if scores else 0
print(f"[GRADE] Average relevance: {avg_score:.2f}")
print(f"[GRADE] Keeping {len(relevant_docs)}/{len(documents)} documents")
return {"documents": relevant_docs, "relevance_score": avg_score}
06:47:09 "The grading is just really prompting our large language model to become a grader."
That is the demystification worth keeping. There is no classifier, no fine-tune, no cross-encoder model. It is a system prompt with a four-point rubric — 1.0 / 0.7 / 0.3 / 0.0 — an instruction to emit nothing but a number, and float() on the result. temperature=0, because a grader that gives different answers to the same pair is not a grader.
One score, two different jobs. The float that comes back is used twice with the same 0.5 threshold but completely different semantics, and conflating them is a common mistake:
- Filtering (per document):
if score >= 0.5: relevant_docs.append(doc). The context you
eventually generate from contains only the survivors. This is a precision filter — Part 6's failure mode #3, retrieval noise, solved at query time instead of index time.
- Routing (aggregate):
avg_scoreover all documents becomesrelevance_score, and the
router compares that against 0.5. This is a health signal for the retrieval attempt as a whole.
The instructor narrates the filter live at 06:56:20: "these two are really high, this is zero, so I'm not going to keep three of them."
3. rewrite — "the brain"
06:47:09 "This is the brain… okay, something is not right. Let's go back and rewrite things."
# fcc-production-rag-part-6/04_agentic_rag.py (lines 209-252)
def rewrite_query(state: RAGState) -> dict:
"""
Rewrite the query to improve retrieval.
Called when initial retrieval doesn't find relevant documents.
"""
query = state["query"]
retry_count = state.get("retry_count", 0)
print(f"\n[REWRITE] Attempt {retry_count + 1}: Improving query...")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.3)
rewrite_prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"""You are a query rewriter for a RAG system.
The original query didn't retrieve relevant documents.
Rewrite the query to be more specific and likely to match relevant documents.
Consider:
- Adding synonyms or related terms
- Being more specific about what information is needed
- Rephrasing to match how documentation is typically written
Output ONLY the rewritten query, nothing else.""",
),
(
"human",
"""Original query: {query}
Rewritten query:""",
),
]
)
chain = rewrite_prompt | llm
result = chain.invoke({"query": query})
rewritten = result.content.strip()
print(f"[REWRITE] Original: '{query}'")
print(f"[REWRITE] Rewritten: '{rewritten}'")
return {"rewritten_query": rewritten, "retry_count": retry_count + 1}
temperature=0.3 — the only non-zero temperature in the entire file. Grading and generation are deterministic; rewriting deliberately is not, because a rewrite that is identical every time is a retry that cannot possibly find anything new.
The rubric has three clauses, and the third is the interesting one: "Rephrasing to match how documentation is typically written." That is a direct, deliberate attack on the embedding-mismatch failure mode — Part 1's failure mode #2, dissected in Part 5: the user writes "how do I cancel", the document says "termination policy", and the vectors never meet. Rewriting is an attempt to translate from question-language into document-language.
This node also owns the termination condition: retry_count + 1. If you forget that increment, your graph loops forever — until LangGraph's own recursion limit aborts the run with an error you will have to go read the traceback to understand.
4. generate — ordinary RAG generation, at the end of a loop
# fcc-production-rag-part-6/04_agentic_rag.py (lines 255-302, prints elided)
def generate_answer(state: RAGState) -> dict:
"""
Generate the final answer using retrieved documents.
"""
query = state["query"]
documents = state["documents"]
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
# Format documents
context = "\n\n".join(
[
f"Source: {doc.metadata.get('source', 'unknown')}\n{doc.page_content}"
for doc in documents
]
)
generate_prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"""You are a helpful assistant answering questions based on provided context.
Use ONLY the information in the context to answer. If the context doesn't contain
enough information, say so clearly.
Always cite your sources by mentioning which document the information came from.""",
),
(
"human",
"""Context:
{context}
Question: {query}
Answer:""",
),
]
)
chain = generate_prompt | llm
result = chain.invoke({"context": context, "query": query})
return {"generation": result.content}
Nothing agentic here — this is Part 1's anti-hallucination prompt pattern doing its job at the end of a longer pipeline. Two details are load-bearing:
Source: <source>is prefixed onto every context block. Citation is only possible if
the source names are physically inside the text the model reads. Sources travel inside the context string, never out of band — same rule as Part 1.
- "Use ONLY the information in the context" + "Always cite your sources." The restriction
clause and the citation instruction. In the recorded run they work: the final answer for query 1 ends (source: langgraph_install.md).
Note that this node generates from state["documents"] — which the grader has already replaced with the filtered survivors. generate never sees the rejected documents.
5. fallback — deliberate, graceful failure
# fcc-production-rag-part-6/04_agentic_rag.py (lines 305-322)
def generate_fallback(state: RAGState) -> dict:
"""
Generate a fallback response when retrieval fails after all retries.
"""
query = state["query"]
print(f"\n[FALLBACK] Retrieval failed after {state.get('retry_count', 0)} attempts")
fallback_message = f"""I couldn't find relevant information to answer your question: "{query}"
This could mean:
1. The information isn't in my knowledge base
2. Try rephrasing your question with different terms
3. The topic might not be covered in the available documents
Would you like to try a different question?"""
return {"generation": fallback_message}
06:55:20 "We are gracefully failing. This is very important."
The single most important property of this node: it is not an LLM call. It is an f-string. That means it costs nothing, adds no latency, and — critically — cannot hallucinate. The one path where you have definitively established that you have no grounding is the last place you want a language model improvising.
It also does two things well as UX: it says why (not in the knowledge base) and what to do (rephrase). Compare that with the alternative the pizza demo would otherwise have produced — an answer about pizza synthesised from three documents about Guido van Rossum.
The router: the crossroads
06:49:13 "Ah, this is the crossroads."
# fcc-production-rag-part-6/04_agentic_rag.py (lines 330-363)
def should_retry_or_generate(
state: RAGState,
) -> Literal["rewrite", "generate", "fallback"]:
"""
Decide whether to retry retrieval or proceed to generation.
This is the BRAIN of agentic RAG - making decisions based on retrieval quality.
"""
relevance_score = state.get("relevance_score", 0)
retry_count = state.get("retry_count", 0)
max_retries = state.get("max_retries", 2)
documents = state.get("documents", [])
print(
f"\n[ROUTER] Evaluating: score={relevance_score:.2f}, retries={retry_count}/{max_retries}, docs={len(documents)}"
)
# If we have relevant documents, generate
if relevance_score >= 0.5 and len(documents) > 0:
print("[ROUTER] -> GENERATE (good relevance)")
return "generate"
# If we can retry, rewrite query
if retry_count < max_retries:
print("[ROUTER] -> REWRITE (low relevance, retrying)")
return "rewrite"
# Out of retries
if len(documents) > 0:
print("[ROUTER] -> GENERATE (out of retries, using available docs)")
return "generate"
else:
print("[ROUTER] -> FALLBACK (no relevant documents)")
return "fallback"
The four branches, in priority order:
| # | Condition | Destination | Reasoning |
|---|---|---|---|
| 1 | relevance_score >= 0.5 and len(documents) > 0 | generate | Good retrieval — answer it |
| 2 | retry_count < max_retries | rewrite | Bad retrieval, budget remains — try a better query |
| 3 | len(documents) > 0 | generate | Out of budget, but something survived the filter |
| 4 | otherwise | fallback | Out of budget with nothing — say so honestly |
Branch 3 is the subtle one, and the instructor calls it out at 06:50:13: "out of retries, we just go straight to generate because we can't retry again."
Degrade before you give up. A mediocre answer from mediocre-but-surviving documents beats no answer — but only after the retry budget is spent, never before it. Branch 3 sits below branch 2 in priority for exactly that reason.
Two observations the chapter does not make explicitly but that matter enormously in production:
The router is pure Python, not an LLM. The chapter keeps calling it "the brain", and the code comment says "This is the BRAIN of agentic RAG." But the judgement lives in grade (an LLM); the decision is deterministic if-logic over state. That split is precisely what makes this loop debuggable, testable and bounded. You can unit-test should_retry_or_generate with a dictionary and no API key. Keep it that way — the moment a model decides your control flow, your control flow becomes unreproducible.
Everything the router needs is already in state. 06:50:13 "all of this has been saved in this brain that is lingering there where all these nodes are able to… tap right into it." The router takes no arguments beyond state and calls nothing. That is the payoff of the state-as-memory design.
Wiring the graph
Nodes exist in isolation until you connect them. 06:51:14 "Once we have those nodes, they are not connected at this point. We need to connect them."
# fcc-production-rag-part-6/04_agentic_rag.py (lines 371-415)
def build_agentic_rag_graph():
"""
Build the LangGraph workflow for agentic RAG.
Flow:
1. retrieve -> grade -> [decision]
2. If low relevance and retries left: rewrite -> retrieve (loop)
3. If good relevance or out of retries: generate
4. If no documents at all: fallback
"""
# Create the graph with our state schema
workflow = StateGraph(RAGState)
# Add nodes
workflow.add_node("retrieve", retrieve_documents)
workflow.add_node("grade", grade_documents)
workflow.add_node("rewrite", rewrite_query)
workflow.add_node("generate", generate_answer)
workflow.add_node("fallback", generate_fallback)
# Set entry point
workflow.set_entry_point("retrieve")
# Add edges
workflow.add_edge("retrieve", "grade")
# Conditional edge from grade
workflow.add_conditional_edges(
"grade",
should_retry_or_generate,
{"rewrite": "rewrite", "generate": "generate", "fallback": "fallback"},
)
# After rewrite, go back to retrieve
workflow.add_edge("rewrite", "retrieve")
# Terminal nodes
workflow.add_edge("generate", END)
workflow.add_edge("fallback", END)
# Compile the graph
app = workflow.compile()
return app
The eleven steps, as a procedure you can transplant to any self-correcting graph:
- Define the state schema as a
TypedDictwith every field any node will read or write. - Write each node as
f(state) -> dict, returning only what changed. - Write the router as a plain function returning a
Literal[...]of destination keys. workflow = StateGraph(RAGState)— pass the class, not an instance.add_node(name, fn)for every node. Nothing is connected yet.set_entry_point("retrieve")— the graph has no default start.add_edge("retrieve", "grade")— an unconditional hop.add_conditional_edges(source, router_fn, {returned_string: destination_node}).add_edge("rewrite", "retrieve")— this single line creates the cycle.add_edge("generate", END)andadd_edge("fallback", END).app = workflow.compile(), thenapp.invoke(initial_state).
Step 9 is the entire lesson compressed to one line:
The cycle is an ordinary edge. There is no special "loop" construct.
add_edge("rewrite", "retrieve")turns a DAG into a graph with a cycle, and everything else — the state, the counters, the router — is the bookkeeping needed to make that cycle terminate.
compile() returns a Runnable with the same interface as any LCEL chain, which means the compiled graph drops into the rest of your stack (FastAPI handler, retry wrapper, tracing callbacks) without special-casing.
The ASCII graph the script prints (lines 485-525) — worth keeping in your own graphs
Original ASCII
┌─────────────────────────────────────────────────────────────┐
│ START │
└─────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────┐
│ RETRIEVE │
│ (get docs from │
│ vectorstore) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ GRADE │
│ (LLM evaluates │
│ relevance) │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
(low relevance, (good relevance) (no docs,
can retry) out of retries)
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ REWRITE │ │ GENERATE │ │ FALLBACK │
│ (improve │ │ (create │ │ (graceful │
│ query) │ │ answer) │ │ failure) │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
│ ▼ ▼
│ ┌─────────────────────────┐
│ │ END │
│ └─────────────────────────┘
│
└──────────► (back to RETRIEVE)
KEY INSIGHT: The loop between REWRITE → RETRIEVE → GRADE
allows the system to self-correct and improve retrieval!LangGraph can also generate this for you: app.get_graph().draw_mermaid() prints Mermaid source, and app.get_graph().draw_mermaid_png() returns PNG bytes — both used throughout langgraph_core.py, conditional_edges.py and cycles_loops.py. Render the real graph in CI and diff it; a hand-drawn diagram goes stale, a generated one cannot.
The instructor's summary of that box, at 06:56:20:
"Between rewrite, retrieve and grade — this is what actually makes the system to be a self-correcting system."
Running it
# fcc-production-rag-part-6/04_agentic_rag.py (lines 445-467)
for query in test_queries:
print("\n" + "=" * 60)
print(f"QUERY: {query}")
print("=" * 60)
# Run the graph
initial_state = {
"query": query,
"rewritten_query": "",
"documents": [],
"generation": "",
"relevance_score": 0.0,
"retry_count": 0,
"max_retries": 2,
"_vectorstore": vectorstore, # Pass vectorstore via state
}
result = app.invoke(initial_state)
print("\n" + "-" * 60)
print("FINAL ANSWER:")
print("-" * 60)
print(result["generation"])
Pre-seed every field. rewritten_query: "", documents: [], generation: "", relevance_score: 0.0, retry_count: 0, max_retries: 2. Nodes that use state["x"] rather than state.get("x") will KeyError on a partially populated state, and TypedDict does not enforce presence — missing keys are simply absent at runtime. Build the graph once and reuse it across queries; only the state is per-request.
The demo corpus is five documents, chosen so that three of them are on-topic and two are pure distractors:
source | topic | Content |
|---|---|---|
langgraph_docs.md | langgraph | LangGraph = stateful multi-actor LLM apps; extends LangChain with cyclic graphs; built-in persistence, streaming, human-in-the-loop |
langgraph_install.md | installation | pip install langgraph>=1.1.0; requires Python 3.10 or higher; works with LangChain 1.x; sync and async |
langgraph_concepts.md | stategraph | StateGraph = core abstraction; nodes process state, edges are transitions; conditional edges branch on state values; graph compiles to an executable workflow |
weather.md | weather | Seattle: rainy winter, mild summer; 40°F in January to 75°F in July; about 37 inches of rain annually |
python_history.md | python | Python created by Guido van Rossum, first released 1991 |
Embedded with text-embedding-3-small into a Chroma collection named agentic_rag_demo. All three LLM roles use gpt-4o-mini.
Three test queries, deliberately chosen to exercise all three exits:
test_queries = [
"How do I install LangGraph?", # Should find relevant docs
"What is StateGraph in LangGraph?", # Should find relevant docs
"How do I make pizza?", # Should NOT find relevant docs (tests fallback)
]
06:57:22 "How do I make pizza? This is going to throw everything off, which is what we need to see."
Query 1 — happy path with a partial filter.
[RETRIEVE] Found 3 documents
1. langgraph_install.md 2. langgraph_install.md 3. langgraph_docs.md
[GRADE] langgraph_install.md: 1.00 | langgraph_install.md: 1.00 | langgraph_docs.md: 0.00
[GRADE] Average relevance: 0.67
[GRADE] Keeping 2/3 documents
[ROUTER] Evaluating: score=0.67, retries=0/2, docs=2
[ROUTER] -> GENERATE (good relevance)
[GENERATE] Creating answer from 2 documents...The final answer names pip install langgraph>=1.1.0, Python 3.10+, LangChain 1.x, and ends (source: langgraph_install.md) — the citation instruction worked.
The interesting number is langgraph_docs.md: 0.00. That document is about LangGraph. It scored zero for an install question because the grader scores answer-relevance, not topic-relevance — it asks "does this contain information that answers this query", not "is this on the same subject". That is exactly the behaviour you want from a grader, and it is a sharper test than the cosine similarity that retrieved the document in the first place.
The grader is a second, semantically stricter filter applied after the vector store's loose one. That is why it catches noise reranking would also catch — and why, if you already run a cross-encoder reranker from Part 6, you should ask whether you need both.
Query 2 — everything relevant. All three hits are langgraph_concepts.md, all score 1.00, nothing is filtered, and the router generates on the first pass:
[GRADE] langgraph_concepts.md: 1.00 ×3
[GRADE] Average relevance: 1.00 | Keeping 3/3 documents
[ROUTER] Evaluating: score=1.00, retries=0/2, docs=3 -> GENERATE (good relevance)The answer paraphrases that document faithfully — nodes as functions that process state, edges as transitions, conditional edges branching on state values.
Query 3 — the full failure path.
attempt 0: [RETRIEVE] Found 3 documents → python_history.md ×3
[GRADE] 0.00, 0.00, 0.00 | Average relevance: 0.00 | Keeping 0/3
[ROUTER] score=0.00, retries=0/2, docs=0 -> REWRITE (low relevance, retrying)
[REWRITE] Attempt 1: Original: 'How do I make pizza?'
Rewritten: 'What are the step-by-step instructions for making homemade pizza,
including dough preparation, sauce recipes, and topping suggestions?'
attempt 1: same 3 docs, 0.00 each, Keeping 0/3
[ROUTER] score=0.00, retries=1/2, docs=0 -> REWRITE (low relevance, retrying)
[REWRITE] Attempt 2: Original: 'How do I make pizza?'
Rewritten: 'What are the step-by-step instructions and essential ingredients for
making homemade pizza from scratch?'
attempt 2: same 3 docs, 0.00 each, Keeping 0/3
[ROUTER] score=0.00, retries=2/2, docs=0 -> FALLBACK (no relevant documents)
[FALLBACK] Retrieval failed after 2 attempts
FINAL ANSWER: I couldn't find relevant information to answer your question: "How do I make pizza?"Read those rewrites carefully. They are good queries. If a pizza corpus existed, either one would retrieve from it. The loop is working exactly as designed — and it still fails, because the knowledge base has no pizza in it. 06:59:23 "This is the self-correcting we're talking about."
And nothing in that trace was orchestrated by hand. 07:01:26 "This is automatic, autonomous, because the agent is able to deal with all of that." Three retrievals, two rewrites, nine grades and a graceful exit came out of one app.invoke(initial_state) call.
The full run cost of that one unanswerable question: 3 retrievals, 9 grading LLM calls, 2 rewrite LLM calls, 3 query embeddings — to produce a fixed string. Which brings us to the bill.
The bill: latency and cost, stated honestly
The chapter is candid that the loop costs money — 07:01:26 "each time these calls are happening, remember you're calling the embedding, you're calling a large language model" — but it never quantifies it. Here is the accounting, derived from the code rather than asserted.
Let k be the retriever's k (3 here) and R the number of retries actually taken (0 to max_retries, default 2).
| Work item | One-shot RAG | This agentic graph |
|---|---|---|
| Query embeddings | 1 | 1 + R |
| Vector searches | 1 | 1 + R |
| Grading LLM calls | 0 | k × (1 + R) |
| Rewrite LLM calls | 0 | R |
| Generation LLM calls | 1 | 1 (or 0 on the fallback path) |
| Total LLM calls | 1 | k(1+R) + R + 1 (one less on the fallback path) |
Substituting the defaults k=3, max_retries=2:
| Scenario | LLM calls | Sequential round-trips before the first answer token |
|---|---|---|
Happy path (R=0) | 3 grades + 1 generate = 4 | 1 retrieval + 3 serial grades (1 if you batch) + generate |
One retry (R=1) | 6 grades + 1 rewrite + 1 generate = 8 | 2 retrievals + 2 grade rounds + 1 rewrite + generate |
Full failure (R=2) | 9 grades + 2 rewrites + 0 generate = 11 | 3 retrievals + 3 grade rounds + 2 rewrites |
Three consequences you should internalise:
- The happy path is already 4× the LLM calls of one-shot RAG, before any retry happens.
Every single query pays for grading, including "hi".
- The failure path is the most expensive path. That inverts the usual economics: in a
one-shot system, a failed retrieval costs the same as a successful one. Here, the queries your system cannot answer are the ones that cost the most. If a meaningful share of your traffic is out-of-corpus, budget for it.
- Grading calls in this implementation are serial. The
for doc in documentsloop makes
k blocking round-trips. Plug your own measured per-call latency into that: whatever a gpt-4o-mini grade costs you, the happy path pays it three times before generation starts, and the failure path is eleven serial LLM round-trips deep before the user sees a fixed string. Measure it in your environment before you decide whether it is tolerable.
The bug in the shipped code
This is the most instructive defect in the whole source repo, so it gets its own section.
Look again at these two lines, in retrieve_documents:
vectorstore = state.get("_vectorstore") # Injected at runtime
if not vectorstore:
# Fallback - create new (in production, pass via config)
vectorstore = create_sample_vectorstore()
and at the initial state:
"_vectorstore": vectorstore, # Pass vectorstore via state
_vectorstore is not declared in the RAGState TypedDict. LangGraph builds its state channels from the schema, so a key that is not in the schema is dropped. state.get("_vectorstore") therefore returns None on every call, the "fallback" branch fires every time, and create_sample_vectorstore() re-embeds and re-inserts all five sample documents. Because Chroma's in-process ephemeral client is cached per settings, every one of those inserts lands in the same agentic_rag_demo collection, which grows monotonically for the life of the process.
The recorded run proves it, and this is the detail that makes the diagnosis airtight:
- Query 1's top-3:
langgraph_install.md,langgraph_install.md,langgraph_docs.md— the
install doc twice.
- Query 2's top-3:
langgraph_concepts.mdthree times. - Query 3's top-3:
python_history.mdthree times.
The corpus contains each of those files exactly once. The duplicate count grows by one per retrieve call, because each call inserts another copy.
Consequences, in ascending order of seriousness:
k=3is returning one unique document dressed up as three.- You pay to embed five documents on every retrieval, which is a hidden per-loop cost the
cost table above does not even include.
relevance_scoreis an average over duplicates. Three copies of one perfect document
average 1.00; one perfect document plus two distractors averages 0.33. The bug distorts routing, which means it distorts the very decisions the chapter is teaching.
The fix is one of two things, and the code comment tells you which one is right:
# Option A (minimal): declare it, so LangGraph keeps it.
class RAGState(TypedDict):
...
_vectorstore: object # or a proper VectorStore type
# Option B (better, and what the comment "in production, pass via config" means):
# don't put a live client in state at all. Close over it, or pass it through
# RunnableConfig, and delete the create-new fallback so a missing store fails LOUDLY.
def make_retrieve_node(vectorstore, k: int = 3):
def retrieve_documents(state: RAGState) -> dict:
query = state.get("rewritten_query") or state["query"]
retriever = vectorstore.as_retriever(search_kwargs={"k": k})
return {"documents": retriever.invoke(query)}
return retrieve_documents
State is for data, not for handles. State gets serialised by checkpointers, diffed by tracers and copied between nodes. A live database client does not belong in it. Inject clients by closure or config; keep state to things you would be happy to see in a JSON log.
And the meta-lesson, which is bigger than this bug: a silent fallback hid a broken injection for an entire recorded lesson. if not vectorstore: build one is exactly the kind of defensive branch that turns a loud AttributeError into a quiet wrong answer. That is invariant 4 again, this time inside your own orchestration code.
The rest of the traps
| # | Trap | Why it bites | What to do |
|---|---|---|---|
| G2 | rewrite always rewrites state["query"], never the previous rewrite | Attempt 2 is an independent re-roll, not a refinement | Feed previous rewrites + their scores into the prompt |
| G3 | One grading LLM call per document, per attempt | k=3, R=2 → 9 grade calls for a failure | Batch, parallelise, short-circuit, or use a reranker |
| G4 | except ValueError: score = 0.5 | A malformed grade lands exactly on the keep threshold | Fail closed at 0.0; log the raw reply |
| G5 | grade overwrites documents with the filtered list | The rejected documents are gone from state forever | Write survivors to a new key; keep the raw set for debugging |
| G6 | Grading always uses the original query, even after a rewrite | A rewrite that finds documents matching the new phrasing can still be graded 0 against the old one | Defensible either way — decide deliberately and comment it |
| G7 | set_entry_point("retrieve") means every query retrieves | "hi" costs a retrieval + k grade calls | Add a real first decision node, or a cheap gate (Part 11) in front of the graph |
| G12 | Dead code in the lesson file: unused import os, unused Annotated and from operator import add, from langchain_chroma import Chroma duplicated on lines 21–22 | Harmless, but it is copy-paste residue | Don't cargo-cult it |
| G13 | vectorstore.delete_collection() only runs on success | An exception mid-demo leaves the collection populated for the next run | try/finally, or a fresh persist_directory per run |
Two more the chapter does not raise but that you will hit:
- The graph has no global timeout.
max_retriesbounds the number of loops, not wall-clock
time. A slow LLM turns a bounded loop into an unbounded latency. Put a deadline on the whole app.invoke and degrade to the fallback string when it trips.
- Nobody evaluates the grader. Nothing in this chapter checks whether
gpt-4o-mini's
0/0.3/0.7/1.0 scores are calibrated against human judgement. Your entire routing logic is built on that model's opinion. Sample fifty (query, document) pairs, label them yourself, and measure agreement before you trust a 0.5 threshold with production traffic.
The same loop, generalised
The RAG version is one instance of a pattern that shows up everywhere once you have a StateGraph. The sibling repo has two more, and comparing them teaches something the RAG version alone cannot.
A quality-refinement loop — same shape, LLM evaluator, score threshold plus an iteration cap:
# production-course-main-code/conditional_edges.py (lines 131-152)
def should_continue(state: QualityState) -> Literal["improve", "finalize"]:
if state["quality_score"] >= 7:
return "finalize"
elif state["iteration"] >= 3:
return "finalize" # Max iterations
else:
return "improve"
graph = StateGraph(QualityState)
graph.add_node("evaluate", evaluate_quality)
graph.add_node("improve", improve_content)
graph.add_node("finalize", finalize_content)
graph.add_edge(START, "evaluate")
graph.add_conditional_edges(
"evaluate", should_continue, {"improve": "improve", "finalize": "finalize"}
)
graph.add_edge("improve", "evaluate") # Loop back!
graph.add_edge("finalize", END)
Identical skeleton: evaluate → route → {improve and loop | finalize}. Note it has the same two-condition escape hatch — quality met or budget exhausted — and note also that iteration is only incremented in improve, so evaluate re-runs its LLM call on every pass. Same cost shape as our grader.
A self-correcting code generator — and this one is more interesting, because its evaluator is not an LLM:
# production-course-main-code/cycles_loops.py (lines 54-93, abridged)
def validate_code(state: CodeGenState) -> dict:
code = state["code"]
# Step 1: Does it compile?
try:
compile(code, "<string>", "exec")
except SyntaxError as e:
return {"errors": [f"SyntaxError: {e}"], "success": False}
# Step 2: Does it RUN and produce correct results?
test_cases = [
([3, 1, 4, 1, 5, 9], 5), # normal case
([1, 1, 1], None), # all same → no second largest
([7], None), # single element
([3, -1, 3, 5, 5], 3), # duplicates at top
]
...
return {"success": True}
And, crucially, its generator node reads the previous error:
# production-course-main-code/cycles_loops.py (lines 31-52, abridged)
def generate_code(state: CodeGenState) -> dict:
if state["iteration"] == 0:
# First attempt
prompt = f"Write Python code for: {state['task']}\nReturn only the code."
else:
# Correction attempt
prompt = (
f"Fix this Python code:\n{state['code']}\n\n"
f"Errors:\n{state['errors'][-1]}\n\n"
"Return only the corrected code."
)
...
return {"code": code, "iteration": state["iteration"] + 1}
That is what the RAG rewrite node is missing (trap G2). The code loop's retry is informed: it knows exactly what failed last time and is told to fix that. The RAG loop's retry is blind: it re-rolls the original query and hopes.
One caveat before you copy this file wholesale: its fixtures have drifted. The task it invokes is "a function that calculates factorial recursively", while validate_code looks for a function named solve and asserts second-largest-element behaviour — a harness that can never pass. demo_self_correcting_code() is commented out in __main__, so nobody noticed. A deterministic evaluator is only as good as its agreement with the task; that agreement is itself something you have to test.
Two rules fall out of this comparison, and they are the most transferable things in this part: 1. Prefer a deterministic evaluator whenever one exists.
compile()plus a handful of test cases costs nothing, never hallucinates, and never needs calibrating. An LLM grader is what you use when — as in relevance judgement — no cheap deterministic check exists. 2. Feed the failure back into the retry. A retry that does not know why the last attempt failed is a slot machine with a token meter attached.
What this lesson leaves out, and what production needs
The demo corpus itself advertises three LangGraph features the lesson then never uses — persistence, streaming and human-in-the-loop. That is not an accident of writing; those are the features that turn this graph into a product.
| Missing piece | Why you need it | Where it lives |
|---|---|---|
| Checkpointing | Resume a long loop after a crash; per-conversation memory; time-travel debugging of a retry | graph.compile(checkpointer=MemorySaver()) or SqliteSaver.from_conn_string(db_path), with config={"configurable": {"thread_id": "user-123"}} — production-course-main-code/checkpointing.py |
| Human-in-the-loop | Interrupt before an expensive or irreversible node and ask a person | production-course-main-code/human_in_loop.py |
| Streaming per node | The user sees "searching… evaluating… rewriting…" instead of a long silence — the single best mitigation for the latency this pattern adds | LangGraph streaming APIs (not covered in the course) |
| Tracing the loop | See which branch every query took, and how many retries | LangSmith, Part 7 |
| Multiple retrieval tools | Pattern 4 — escalate strategy, not just phrasing | production-course-main-code/advanced_rag.py: MultiQuery, ContextualCompression, Ensemble(BM25), ParentDocument retrievers |
| Multi-step decomposition | Pattern 3 — split a compound question into sub-queries and retrieve for each | Asserted at 06:41:44, implemented nowhere |
| Grader evals | Nobody validates the grader's calibration | The evals pillar, Part 7 |
Of those, streaming per node deserves special emphasis. Agentic RAG's honest cost is latency, and the cheapest way to make a slow answer tolerable is to show the user the loop working. LangGraph gives you node-level streaming for free once your control flow is a graph — which is, circularly, one of the better reasons to have built it as a graph.
When to use it — and when not
The script prints its own six takeaways at 07:02:28, and they are a fair summary of everything above:
- Traditional RAG is one-shot: retrieve → generate
- Agentic RAG adds evaluation and retry loops
- LangGraph
StateGraphenables cyclic workflows - Key pattern: retrieve → grade → [rewrite if needed] → generate
- Graceful fallback when retrieval fails completely
- This is THE production RAG pattern for 2026
On #3, the instructor's own gloss at 07:02:28: cyclic workflows mean the graph "can go and retry, go back and forth until something is met — then it breaks out of the loop." The "until something is met" is doing all the work in that sentence, and it is what max_retries guarantees.
The four criteria are printed by the script itself at 07:03:30, with the instructor's gloss:
| Criterion | What it means in practice |
|---|---|
| Complex queries that might need reformulation | Users write questions in their own vocabulary, your documents use a different one |
| High-stakes applications where answer quality matters | "Let's say you're building on top of legal documents or things that need to be specific" — legal, compliance, medical |
| When you have diverse document types | "Legal documents, guides of some sort, instructions — all those different kinds of documents" — one retrieval strategy will not suit all of them |
| User-facing applications (vs batch processing) | A human is waiting and will judge the answer; in batch, nobody sees the failure in real time and you can retry offline for free |
And the counter-case, stated just as plainly 07:03:30: "If you're building something very simple — obviously we're just testing things here — you don't really need agentic RAG."
A decision matrix that puts both together:
| Your situation | Verdict |
|---|---|
| Homogeneous corpus, users share its vocabulary, retrieval quality already measured good | One-shot. The loop will almost never fire and you pay for grading on every query anyway |
| Latency-sensitive (chat with a strict p95, autocomplete, voice) | One-shot, or agentic with node streaming and max_retries=1 |
| Batch / offline pipelines | One-shot per pass; retry at the job level, where retries are cheap |
| Failures are coverage gaps (the answer genuinely is not in the corpus) | Fix the corpus. The loop makes those failures more expensive, not fewer |
| Failures are vocabulary gaps (the answer is in there, phrased differently) | Agentic — this is exactly what it fixes |
| High-stakes, user-facing, diverse document types | Agentic, with a strict budget and a fallback |
| You need "I don't know" to be reliable | Agentic — the grade + fallback pair is the cleanest way to build a trustworthy refusal |
The chapter closes on the meta-lesson repeated all course long, and it is the correct note to end on:
07:03:30 "It's always about knowing when to use and when not to use."
Restated one last time at 07:04:31, just before the course turns to GraphRAG (Part 16): if you are in production, look at your use case and decide whether it actually calls for this.
Checklist — decisions to make before you ship a self-correcting loop
Architecture
- Is your failure mode vocabulary or coverage? Sample 30 failed queries and label them. If most are coverage gaps, close them and skip this entire part.
- Where does the evaluation signal come from — an LLM grader, a cross-encoder reranker score, or a deterministic check? Prefer deterministic; prefer cheap; use an LLM last.
- Does your graph retrieve unconditionally (
set_entry_point("retrieve")) or does it get a real first decision? At minimum, gate trivial inputs before the graph. - Route on the mean of the scores, the max, or "at least one document ≥ X"? Decide and write down why. The course's mean is a choice, not a law.
Budgets
- Set
max_retriesexplicitly, per request, in state. Default2. Never unbounded. - Set a wall-clock deadline for the whole
app.invoke, independent of the retry count. - Compute your worst-case call count:
k(1+max_retries) + max_retries + 1LLM calls. Atk=3andmax_retries=2that is 12. Multiply by your traffic. Decide whether you can afford the failure path, because it is your most expensive path. - Batch or parallelise the grading calls before you ship. Serial
kgrading round-trips on every query is the default, and it is the wrong default.
Correctness
- Declare every field you put in state — including injected handles — or inject them by closure/config instead. Delete "create it if missing" fallbacks; fail loudly.
- Fail the grade parser closed (
0.0), not open (0.5), and log every unparseable reply. - Keep the raw retrieval set in state alongside the filtered one, so you can debug what was thrown away.
- Feed the previous rewrite and its score into the next rewrite prompt. A blind retry is a coin flip you paid for.
- Decide whether grading uses the original or the rewritten query, and comment the decision in the code. Both are defensible; silence is not.
Operations
- Build the graph once at startup; only the state is per request.
- Emit structured logs from every node with a request id, so a three-attempt run reconstructs from logs alone. Replace the
printstatements. - Track retry-rate, fallback-rate, grade-calls-per-query and p95 end-to-end latency. If retry-rate is near zero, you are paying for a loop that never fires — go back to one-shot.
- Stream node transitions to the user. It is the cheapest fix for the latency this pattern costs.
- Test the failure path in CI: a query your corpus cannot answer must reach
fallback, and must never reachgenerate. - Calibrate the grader against human labels before trusting the
0.5threshold. - Render the graph (
app.get_graph().draw_mermaid()) in CI and diff it. Generated diagrams cannot go stale.
The one-sentence version
A self-correcting retrieval loop is worth its bill exactly when your system's failures are recoverable by asking a better question. Measure that first; build the graph second.