oriolrius.me Production RAGReference guide
Part 16·Advanced & frontier

The Frontier: GraphRAG, Multimodal RAG and Where This Is Going

GraphRAG and multimodal ColPali retrieval: when a document is not just text.

46 min read 26 sections Advanced & frontier
Audio summary · 5:13 · EN
The Frontier: GraphRAG, Multimodal RAG and Where This Is Going

Everything in Parts 1–15 shares one assumption so deep it never had to be stated: a document is text, and relevance is a distance between two vectors. Chunking, hybrid search, reranking, contextual retrieval, agentic self-correction — all of it is a fight to get the right span of characters into the top-k. Every technique in this guide so far improves the aim. None of them changes the target.

This part covers the two techniques that change the target, and they break the assumption in opposite directions:

  • GraphRAG rejects the idea that relevance is a score. It materialises exact, symbolic relations between entities and retrieves by walking them. It answers a class of question that vector search cannot answer at any k, with any embedding model, at any budget.
  • Multimodal RAG (ColPali) rejects the idea that a document is text. It renders each page to an image, embeds the picture, and hands retrieved page images to a model with eyes. It recovers information your pipeline is currently destroying at ingest, before retrieval ever runs.

Then a closing synthesis: what the field's four eras actually were, which of the six advanced techniques are floor versus edge, and — stated plainly — which parts of this frontier the source material asserts but never demonstrates.

In the two-phase mental model from Part 1, here is exactly which boxes each one modifies:

TechniquePhase A (index)Phase B (query)Budget hit
GraphRAGAdds LLM entity/relation extraction, graph construction, community detection, community summaries — after Load, replacing Split→Embed as the primary indexReplaces Retrieve with entity identification + traversal; keeps vector search alongside itMoney and latency, at index time
Multimodal / ColPaliDeletes Split entirely; Embed becomes a GPU vision model over rendered pagesRetrieve returns page images; Generate must be a vision LLMMoney, at query time (~10×)

Both are invariant-3 stories in reverse. GraphRAG pushes an enormous amount of work left into the one-time index — which is why it is disqualified for real-time corpora. Multimodal pushes cost right, onto every single request — which is why it is disqualified for high-traffic, cost-sensitive apps. Neither is free, and neither is a default.

1 · GraphRAG: the questions vector search structurally cannot answer

GraphRAG was introduced by Microsoft as a corrective to vector-only retrieval 07:04:45. The framing matters: it is not a replacement for RAG, and — despite how the name reads — it is not "graphs instead of vectors." It is a different retrieval mechanism stapled onto the same pipeline, for a specific class of query.

The failure mode: chunks are islands

Standard RAG retrieves isolated chunks. The chunks that come back are not correlated; there is no relation between them, no edge, nothing connecting fact A in chunk 3 to fact B in chunk 47. Each retrieved chunk is an island, independently scored, and the top-k is a bag of islands 07:04:45.

That is completely fine as long as every fact an answer needs lives inside one chunk. The moment the answer is a composition of facts living in different chunks, vector search has no mechanism to compose them. There is no join operator in cosine similarity.

The instructor's motivating query 07:04:45:

"Who are the competitors of companies that John advises?"

Watch what happens. The retriever surfaces chunks about John. It surfaces chunks about competitors. Both halves of the answer are in the context window. And the system still fails — because the join between them was never materialised anywhere. It "cannot connect John with the competitors at all."

With a graph, the same query walks instead of scoring: John → ADVISES → TechCorp → COMPETES_WITH → DataTech Inc. The output is not just "DataTech Inc." but the chain that produced it — what the instructor calls answering "with actual reasoning path" 07:05:47. That derivation is the deliverable, and Part 1's trust chain (sources → citations → verifiability) is why: an answer you cannot audit is an answer you cannot ship.

The crux, in six words

The course's own demo makes the mechanism painfully precise. Corpus of five sentences, one query — this is real code from the lesson file:

Python
# fcc-production-rag-part-6/05_graphrag_intro.py — demonstrate_multihop_problem(), lines 41-49
documents = [
    "John Smith is the CEO of TechCorp. He joined the company in 2015.",
    "Sarah Johnson is John Smith's executive assistant. She manages his calendar.",
    "Sarah Johnson works in the Executive department on the 5th floor.",
    "The Executive department also includes Mike Brown and Lisa Chen.",
    "Mike Brown is the Chief Financial Officer. Lisa Chen is the Chief Legal Officer.",
]

query = "Who works in the same department as the CEO's assistant?"

Answering that requires four hops 07:12:58:

  1. Find the CEO → John Smith.
  2. Find the CEO's assistant → Sarah Johnson.
  3. Find Sarah's department → Executive.
  4. Find others in that department → Mike Brown, Lisa Chen (and John himself).

Vector search might well retrieve documents 3 and 4, because they contain the word "department." It will never make hop 1→2→3, and the reason is the single most transferable sentence in the chapter 07:13:59:

"Sarah's name isn't in the query."

Embedding similarity can only match on meaning that is present in the query text. The bridging entity is unknown at query time — by definition, because finding it is the question. No embedding model can retrieve a chunk keyed on a name the query does not contain. This is not a quality problem you can tune your way out of; it is a structural property of nearest-neighbour retrieval.

The lesson names three conditions under which GraphRAG is "particularly powerful" 07:09:52, and they are worth reading as a conjunction rather than a menu: complex questions requiring reasoning, datasets with many interconnected entities, and —

"Questions about relationships, not just facts." 07:09:52

That last one is the one-line test. If you cannot point at real traffic that asks about relationships, the other two do not matter.

The method: four steps

GraphRAG's mechanism, as taught [07:06:49–07:09:52]:

  1. Extract entities and relationships from documents. Entities are "the nouns, the people, the things" — a person, a company, a product, a place, a concept. Relationships are how those entities connect. An LLM does this pass, though rules/NER/manual curation are alternatives the course mentions and never enumerates 07:17:01.
  2. Build the knowledge graph. Entities become nodes, relationships become edges. The instructor is emphatic that this is a graph in the mathematical sense — vertices and edges, not a metaphor 07:07:51.
  3. Traverse for multi-hop reasoning. From any node you hop along edges to reach facts no single chunk contained.
  4. Hybrid retrieval — combine graph traversal with vector search to produce the final answer. "Step four is the magic" 07:09:52.

Step 4 is the one people forget. GraphRAG is not graph-instead-of-vectors; the production shape is graph traversal plus vector search feeding one answer. Treating the graph as a wholesale replacement for your vector store throws away recall on every non-relational query in your traffic.

If you have read Part 15, step 2 will feel familiar, and the instructor says so out loud while adding edges 07:11:56:

"This is very similar to LangGraph, right? Amazing — same thing, we're using nodes, same concept. Thank you, math."

That aside is worth internalising. A LangGraph agent graph and a knowledge graph are the same underlying object — nodes plus edges. Only the semantics of a node differ: a step in a workflow versus an entity in the world.

Building the graph: real code

The demo uses NetworkX, "because it has all the pieces for creating simple, complex networks" 07:10:55 — uv add networkx, and nothing else. Six nodes, six edges, a directed graph:

Python
# fcc-production-rag-part-6/05_graphrag_intro.py — build_knowledge_graph(), lines 97-124
    # Create a directed graph
    G = nx.DiGraph()

    # Add entities (nodes)
    entities = [
        ("John Smith", {"type": "Person", "role": "CEO"}),
        ("Sarah Johnson", {"type": "Person", "role": "Executive Assistant"}),
        ("Mike Brown", {"type": "Person", "role": "CFO"}),
        ("Lisa Chen", {"type": "Person", "role": "CLO"}),
        ("TechCorp", {"type": "Organization"}),
        ("Executive Department", {"type": "Department", "floor": "5th"}),
    ]

    for entity, attrs in entities:
        G.add_node(entity, **attrs)

    # Add relationships (edges)
    relationships = [
        ("John Smith", "TechCorp", {"relation": "CEO_OF"}),
        ("Sarah Johnson", "John Smith", {"relation": "ASSISTANT_TO"}),
        ("Sarah Johnson", "Executive Department", {"relation": "WORKS_IN"}),
        ("Mike Brown", "Executive Department", {"relation": "WORKS_IN"}),
        ("Lisa Chen", "Executive Department", {"relation": "WORKS_IN"}),
        ("John Smith", "Executive Department", {"relation": "WORKS_IN"}),
    ]

    for source, target, attrs in relationships:
        G.add_edge(source, target, **attrs)

Three things are load-bearing in that listing, and only one of them is obvious.

Nodes carry typed attributes, not just names. {"type": "Person", "role": "CEO"} is what makes hop 1 possible at all — the traversal finds the CEO by filtering on an attribute, not by matching a string. {"type": "Department", "floor": "5th"} shows the same slot carrying arbitrary metadata. This is the graph equivalent of Part 4's lesson that filterability is decided at index time: whatever attribute you did not extract, you cannot traverse on.

Edges are typed. CEO_OF, ASSISTANT_TO, WORKS_IN — screaming-snake-case relation labels. The traversal filters on the exact relation string, so vague or free-form relation labels silently break every hop that depends on them. This is why the extraction prompt later insists on specific relationship types.

Edges are exact facts, and that is the whole point 07:15:01:

"These relationships that you see here are exact relationships that are contained structurally in the graph."

A similarity score is a fuzzy, probabilistic guess about whether two texts are about the same thing. A graph edge is a deterministic, symbolic assertion. Multi-hop works because you can chain deterministic assertions without compounding error; chaining similarity scores compounds error until the result is noise.

Two pieces of loose terminology in the narration are worth not repeating. The instructor calls this structure "a tree" 07:15:01 — it is a directed graph, and directed graphs can and do contain cycles; this one merely happens to look tree-shaped. And at 07:08:52 he describes hopping backwards along edges ("if we're here we can go back here") on a DiGraph, where reverse traversal needs explicit handling — in_edges rather than out_edges, as the rewrite below does. Both slips are harmless in a demo and expensive in a schema.

Traversing it

Four hops, four loops. This is the code as shipped:

Python
# fcc-production-rag-part-6/05_graphrag_intro.py — traverse_graph_for_answer(), lines 159-175
    # Step 1: Find the CEO
    print("\nStep 1: Find the CEO")
    ceo = None
    for node, attrs in G.nodes(data=True):
        if attrs.get("role") == "CEO":
            ceo = node
            print(f"  Found: {ceo}")
            break

    # Step 2: Find the CEO's assistant
    print("\nStep 2: Find the CEO's assistant")
    assistant = None
    for source, target, attrs in G.edges(data=True):
        if target == ceo and attrs["relation"] == "ASSISTANT_TO":
            assistant = source
            print(f"  Found: {assistant} --[ASSISTANT_TO]--> {ceo}")
            break

Steps 3 and 4 are the same shape: scan G.edges(data=True) for source == assistant and relation == "WORKS_IN" to get the department, then scan again for target == department and relation == "WORKS_IN", excluding the assistant, to collect the coworkers.

Full traversal, steps 3–4 and the composed answer (lines 177–200)
Python
# fcc-production-rag-part-6/05_graphrag_intro.py
    # Step 3: Find the assistant's department
    print("\nStep 3: Find the assistant's department")
    department = None
    for source, target, attrs in G.edges(data=True):
        if source == assistant and attrs["relation"] == "WORKS_IN":
            department = target
            print(f"  Found: {assistant} --[WORKS_IN]--> {department}")
            break

    # Step 4: Find others in that department
    print("\nStep 4: Find others in the same department")
    coworkers = []
    for source, target, attrs in G.edges(data=True):
        if target == department and attrs["relation"] == "WORKS_IN":
            if source != assistant:  # Exclude the assistant themselves
                coworkers.append(source)
                print(f"  Found: {source} --[WORKS_IN]--> {department}")

    print("\n" + "-" * 60)
    print("ANSWER:")
    print("-" * 60)
    print(f"The CEO's assistant is {assistant}.")
    print(f"{assistant} works in the {department}.")
    print(f"Others in the same department: {', '.join(coworkers)}")

Output at 07:16:01: the assistant is Sarah Johnson, she works in the Executive Department, and the others there are John Smith, Mike Brown and Lisa Chen. Note that the answer carries its own derivation — the reasoning path — which is what the instructor means by "full context that comes out" 07:05:47.

The instructor flags this immediately, and you should keep the flag 07:11:56:

"Remember, this is just simulation here… this is just to show you how knowledge graph work."

The traversal is didactic scaffolding, not a retriever. It is hardcoded to exactly one question; nothing parses natural language into a graph pattern. But the shipped code also has four defects that are worth naming, because they are the defects your first graph retriever will have:

  1. No not-found handling. If no node has role == "CEO", ceo stays None, every subsequent filter matches nothing, and the function cheerfully prints The CEO's assistant is None. It never raises. This is invariant 4 in its purest form — a graph retriever fails as silently as a vector one.
  2. break on first match everywhere. The code assumes exactly one CEO and exactly one assistant. Real org charts have neither property.
  3. attrs["relation"] will KeyError on any edge added without a relation key — including by your LLM extractor on a bad day. Use attrs.get("relation").
  4. O(E) per hop. Each step scans every edge in the graph. At six edges this is invisible; at six million it is your latency budget.

Here is the idiomatic rewrite. NetworkX gives you incident-edge views directly, which is both faster and more readable — this is a correction to the source, not code from it:

Python
# Idiomatic NetworkX rewrite of the four hops — not from the course repo.
# Uses incident-edge views (O(degree) per hop) instead of full edge scans (O(E)).
import networkx as nx


def find_coworkers_of_ceos_assistant(G: nx.DiGraph) -> dict | None:
    ceos = [n for n, a in G.nodes(data=True) if a.get("role") == "CEO"]
    if not ceos:
        return None            # signal not-found; never fabricate "None" into an answer
    ceo = ceos[0]

    # who points AT the ceo with ASSISTANT_TO?
    assistants = [s for s, _t, a in G.in_edges(ceo, data=True)
                  if a.get("relation") == "ASSISTANT_TO"]
    if not assistants:
        return None
    assistant = assistants[0]

    departments = [t for _s, t, a in G.out_edges(assistant, data=True)
                   if a.get("relation") == "WORKS_IN"]
    if not departments:
        return None
    department = departments[0]

    coworkers = [s for s, _t, a in G.in_edges(department, data=True)
                 if a.get("relation") == "WORKS_IN" and s != assistant]

    return {
        "assistant": assistant,
        "department": department,
        "coworkers": coworkers,
        # the reasoning path is the product, not a debug artefact
        "path": [f"{assistant} --[ASSISTANT_TO]--> {ceo}",
                 f"{assistant} --[WORKS_IN]--> {department}"]
                + [f"{c} --[WORKS_IN]--> {department}" for c in coworkers],
    }

Automating extraction: the piece that makes it scale

You do not hand-author nodes in production. An LLM reads each document and emits entities and edges. This is the extraction prompt, verbatim from the lesson:

Python
# fcc-production-rag-part-6/05_graphrag_intro.py — extract_entities_with_llm(), lines 218-246
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

    extraction_prompt = ChatPromptTemplate.from_messages(
        [
            (
                "system",
                """You are an expert at extracting knowledge graph elements from text.

Given a text, extract:
1. ENTITIES: People, organizations, places, concepts
2. RELATIONSHIPS: How entities are connected

Output in this exact format:
ENTITIES:
- [TYPE] Name: description

RELATIONSHIPS:
- Source --[RELATION]--> Target

Be specific about relationship types (WORKS_FOR, MANAGES, LOCATED_IN, etc.)""",
            ),
            (
                "human",
                """Extract entities and relationships from this text:

{text}""",
            ),
        ]
    )

Four instructions, each doing a job: name the task, enumerate the two output classes, pin an exact output format, and demand specific relation types. That last clause is not style advice — traversal filters on the exact relation string, so WORKS_FOR and "works for, kind of" are different edges and one of them is unreachable.

The demo input is an Acme Corporation press release (lines 248–254) mentioning Jennifer Lee appointed CTO, reporting to CEO Marcus Chen, previously leading AI research at DataTech Inc. in Boston, with HQ in San Francisco and an engineering team of 50 developers. The model emits typed entities — Organization (Acme, DataTech), Person (Jennifer Lee, Marcus Chen), Location (Boston, San Francisco) — and then, critically, Concepts: "Chief Technology Officer," "research team," "engineering team" [07:17:01–07:18:02].

Concepts are first-class entities. A knowledge graph is not a people-and-places directory; roles, teams, policies and product lines all become nodes you can traverse.

The chain itself is two lines:

Python
# fcc-production-rag-part-6/05_graphrag_intro.py — lines 259-266
    chain = extraction_prompt | llm
    result = chain.invoke({"text": sample_text})

    print(f"\nExtracted knowledge graph elements:")
    print("-" * 60)
    print(result.content)

    return result.content

The fix for (2) is the one that matters, and it is the difference between a demo and an ingest pipeline: do not ask an LLM to format text, ask it for a schema. Use structured output over a Pydantic model so the extractor returns objects you can feed straight into G.add_node / G.add_edge:

Python
# Structured-output version of the extractor — not from the course repo.
# Replaces free-text formatting instructions with a schema the model must satisfy.
import networkx as nx
from langchain_openai import ChatOpenAI
from pydantic import BaseModel, Field

class Entity(BaseModel):
    name: str
    type: str = Field(description="Person | Organization | Location | Concept | Department")
    description: str = ""

class Relationship(BaseModel):
    source: str
    relation: str = Field(description="SCREAMING_SNAKE_CASE, e.g. WORKS_FOR, MANAGES, LOCATED_IN")
    target: str

class GraphChunk(BaseModel):
    entities: list[Entity]
    relationships: list[Relationship]

extractor = ChatOpenAI(model="gpt-4o-mini", temperature=0).with_structured_output(GraphChunk)

def ingest(doc_text: str, G: nx.DiGraph) -> None:
    out = extractor.invoke(f"Extract entities and relationships from:\n\n{doc_text}")
    for e in out.entities:
        G.add_node(e.name, type=e.type, description=e.description)
    for r in out.relationships:
        G.add_edge(r.source, r.target, relation=r.relation)

That version has a defect the course never mentions and which will be the first thing that breaks on your corpus: entity resolution. Across ten documents your extractor will produce "John Smith", "J. Smith", "Mr. Smith" and "the CEO" as four separate nodes, and your traversal will find none of them connected. Deduplicating and canonicalising entities — by embedding similarity, by alias tables, by an LLM merge pass — is a required stage of any real GraphRAG ingest and is not covered anywhere in the source material. Budget for it.

The full architecture: what production GraphRAG actually does

The demo covers extraction → graph → traverse. Real GraphRAG has two more indexing stages, both printed in the lesson's ASCII architecture [07:18:02–07:19:03]:

Indexing phase

  1. Ingest documents.
  2. LLM extraction → entities and relations.
  3. Build the knowledge graph.
  4. Community detection — cluster the entity set into sub-groups of densely related entities. The threshold the instructor gives for when this becomes worth doing: "the entities could be 100 entities" 07:19:03. A large flat entity set is unusable holistically; communities give you a coarse index over the graph.
  5. Generate community summaries — an LLM writes a summary of each detected community.

Query phase

  1. Identify the entities mentioned in the incoming query. (This is what anchors a natural-language question into the graph, and it is exactly the step the hand-written demo skips.)
  2. Traverse the graph from those entities.
  3. Generate the response from the traversed subgraph — or, for theme questions, from the community summaries.

Which leads to the distinction that defines GraphRAG, printed verbatim in the lesson file:

text
# fcc-production-rag-part-6/05_graphrag_intro.py — show_graphrag_architecture(), lines 322-332
    TWO QUERY MODES:

    LOCAL SEARCH (Multi-hop reasoning):
    - Identify entities in query
    - Traverse relationships to find answers
    - Good for: specific questions about connections

    GLOBAL SEARCH (Holistic understanding):
    - Use community summaries
    - Aggregate knowledge across documents
    - Good for: "What are the main themes?" type questions
Local searchGlobal search
MechanismIdentify entities → traverse edgesRead community summaries → aggregate
ReadsThe graphThe summaries built at index time
Answers"Who works with the CEO's assistant?""What are the main themes across these documents?"
Also calledMulti-hop reasoning 07:19:03Holistic understanding 07:20:05

Point questions traverse edges; theme questions read summaries. That sentence is the single most transferable idea in the GraphRAG chapter, and it generalises past GraphRAG: any corpus-level question ("what changed this quarter", "what are people complaining about") is a summarisation problem wearing a retrieval costume, and top-k retrieval is the wrong tool for it regardless of your stack.

The chapter never shows a router that picks local versus global per query. In practice you either expose the mode as an API parameter, or you classify the query with a cheap model — the same shape as the multimodal router discussed later in this part.

Implementation options

The lesson prints four; the instructor narrates two and skips past the other two [07:20:05–07:21:06]. All four are worth having, because option 4 is the concrete recipe behind the "step four is the magic" hybrid claim.

OptionShapeProsCons
1. Microsoft GraphRAGpip install graphrag, CLI-driven index + queryFull-featured, well-documented, production-readyHeavy; requires significant compute for indexing
2. LangGraph + Neo4jNeo4jGraph + GraphCypherQAChainFull control, integrates with an existing Neo4jYou manage Neo4j; manual entity extraction
3. LlamaIndex KnowledgeGraphIndex (on screen, never narrated).from_documents(..., max_triplets_per_chunk=10, include_embeddings=True)Easy to use, good LlamaIndex integration"Less powerful than full GraphRAG"
4. Hybrid vector + graph (on screen, never narrated)vector search → extract entities from the hits → traverse → LLM synthesises"Best of both worlds"More complex to implement

Option 1 is the instructor's recommendation, twice [07:20:05, 07:23:07]:

"Microsoft GraphRAG is the go-to production implementation."

Its whole surface, as shown, is three commands — and note that the local/global split is literally a CLI flag:

bash
# fcc-production-rag-part-6/05_graphrag_intro.py — show_implementation_options(), lines 356-363
pip install graphrag

# Index documents
graphrag index --root ./my_docs

# Query
graphrag query --method local "Who works with the CEO?"
graphrag query --method global "What are the main themes?"
Options 2–4 as printed in the lesson file (lines 369–416), with corrections
Python
# fcc-production-rag-part-6/05_graphrag_intro.py — OPTION 2: LangGraph + Neo4j
from langchain_community.graphs import Neo4jGraph
from langchain.chains import GraphCypherQAChain

graph = Neo4jGraph(url="bolt://localhost:7687")

chain = GraphCypherQAChain.from_llm(
    llm=ChatOpenAI(),
    graph=graph,
    verbose=True
)
result = chain.invoke("Who reports to the CEO?")
Python
# fcc-production-rag-part-6/05_graphrag_intro.py — OPTION 3: LlamaIndex
from llama_index.core import KnowledgeGraphIndex

kg_index = KnowledgeGraphIndex.from_documents(
    documents,
    max_triplets_per_chunk=10,
    include_embeddings=True
)
query_engine = kg_index.as_query_engine()
response = query_engine.query("Who works in Engineering?")

OPTION 4 — Hybrid vector + graph (prose in the file, lines 405–416):

  1. Vector search to find relevant documents.
  2. Extract entities from the retrieved docs.
  3. Graph traversal for multi-hop reasoning.
  4. LLM synthesises the final answer.

Corrections you need before running any of this:

  • from langchain_community.graphs import Neo4jGraph and from langchain.chains import GraphCypherQAChain have both moved. The maintained home is the langchain-neo4j package: from langchain_neo4j import Neo4jGraph, GraphCypherQAChain.
  • Neo4jGraph(url="bolt://localhost:7687") passes no credentials. The constructor falls back to the NEO4J_USERNAME / NEO4J_PASSWORD environment variables and raises if neither those nor explicit arguments are present — so the line only "works" on a machine with the right env already exported. Pass them explicitly, or you will ship a config dependency you never wrote down.
  • Security: GraphCypherQAChain generates and executes Cypher derived from user input. Current LangChain requires you to opt in with allow_dangerous_requests=True for exactly this reason. Point it only at a read-only Neo4j account — this is prompt injection with a database behind it (Part 12).
  • LlamaIndex has since steered users from KnowledgeGraphIndex toward PropertyGraphIndex.

None of those four corrections come from the course; verify each against the current library docs before you commit to a version.

Note also that requirements.txt pins pyvis>=0.3.2 — presumably to render the graph — and no file in the repo ever imports it. Visualising your extracted graph is the cheapest debugging tool you will find for an extraction pipeline; the dependency is already there.

When to use it, and when it is over-engineering

The lesson prints the decision as two four-item lists [07:21:06–07:22:07]:

✅ Use GraphRAG when❌ Skip GraphRAG when
Documents describe relationships (org charts, research papers)Simple fact retrieval — use standard RAG
Queries need multi-hop reasoningSmall document sets (< 100 docs)
"Who / what is connected to X?" questionsReal-time indexing requirements
You need global summarization across documentsCost-sensitive applications (indexing is expensive)

The real-time disqualifier deserves unpacking, because the chapter states it without explaining it. GraphRAG's index is not append-friendly: a new document can introduce entities that belong in existing communities, which invalidates those communities' summaries, which are what global search reads. Adding one document can therefore require re-running community detection and re-summarising. Combine that with one LLM call per chunk for extraction, and "indexing is expensive" stops being a vague warning and becomes a specific, recurring bill. The demo graph, for its part, lives in memory in NetworkX for the life of the process — persistence, incremental updates and re-indexing are never covered.

The lesson closes with seven takeaways, printed in __main__ (lines 466–476); they compress to five that are worth memorising:

1. Traditional RAG fails at multi-hop reasoning. 2. GraphRAG builds knowledge graphs from documents. 3. Entities (nodes) + relationships (edges) = traversable structure. 4. Local search traverses the graph for specific answers; global search uses community summaries for themes. 5. Use it when relationships matter and multi-hop queries are expected — "anything less than that, then you don't need to implement GraphRAG" 07:23:07.

2 · Multimodal RAG: when the meaning is in the pixels

The bridge into this material is one sentence 07:24:09:

"The world is not just text."

Every technique in this guide so far — chunking, hybrid search, reranking, contextual retrieval, GraphRAG — assumes the corpus is text. Real documents are PDFs with tables, charts, scans, diagrams. A pipeline whose first step is "extract the text" has already decided that everything non-textual is noise, and it makes that decision at ingest, before retrieval or generation ever runs. No amount of reranking or prompt engineering downstream recovers it.

ColPali is defined in one clause 07:24:31:

"ColPali is a vision language model that creates embeddings from document images, not extracted text."

The whole technique is an unpacking of that sentence: the unit of indexing changes from a chunk of characters to a picture of a page.

The demonstration: extraction is a lossy operation

The lesson prints a table, and then prints what PyPDF or pdfminer typically make of it. Both are real strings in the source file:

Python
# fcc-production-rag-part-6/06_multimodal_rag.py — demonstrate_extraction_problem(), lines 44-64
    # Simulated table in a PDF
    original_table = """
    ┌─────────────────────────────────────────────────────────┐
    │              Q1 2025 Sales by Region                    │
    ├─────────────┬─────────────┬─────────────┬───────────────┤
    │ Region      │ Q1 Target   │ Q1 Actual   │ Variance      │
    ├─────────────┼─────────────┼─────────────┼───────────────┤
    │ North       │ $2.5M       │ $2.8M       │ +12% ✓        │
    │ South       │ $1.8M       │ $1.5M       │ -17% ✗        │
    │ East        │ $3.2M       │ $3.4M       │ +6% ✓         │
    │ West        │ $2.1M       │ $2.0M       │ -5%           │
    ├─────────────┼─────────────┼─────────────┼───────────────┤
    │ TOTAL       │ $9.6M       │ $9.7M       │ +1%           │
    └─────────────┴─────────────┴─────────────┴───────────────┘
    """

    # What text extraction often produces
    extracted_text = """
    Q1 2025 Sales by Region Region Q1 Target Q1 Actual Variance
    North $2.5M $2.8M +12% South $1.8M $1.5M -17% East $3.2M $3.4M +6%
    West $2.1M $2.0M -5% TOTAL $9.6M $9.7M +1%
    """

Exactly four things died in that transformation [07:26:51–07:27:04]:

  1. Table structure — rows and columns are jumbled into one run-on line.
  2. Visual indicators — the ✓ and ✗ symbols may be dropped entirely.
  3. Alignment — you can no longer tell which number belongs to which region.
  4. Context — "South is underperforming" is visually obvious on the page and completely absent from the string.

The lesson's banner for this is blunt, and it is the correct mental label: "TEXT EXTRACTION IS A LOSSY OPERATION!"

Now escalate 07:27:14. Complex financial reports with nested tables. Charts showing trends. Diagrams with arrows and connections. Flowcharts and org charts. On any of those, a text-first pipeline is not merely suboptimal — it is discarding the majority of the information on the page and then confidently answering from the remainder.

The two pipelines

The lesson draws them side by side, and the diagram is the clearest artefact in the file:

Figure 1Text-extraction RAG vs ColPali vision RAG, and how vision RAG answers a query
Original ASCII
# fcc-production-rag-part-6/06_multimodal_rag.py — explain_colpali_approach(), lines 109-139
    TRADITIONAL RAG PIPELINE:
    ┌─────────┐    ┌──────────────┐    ┌─────────────┐    ┌───────────┐
    │   PDF   │───▶│ Extract Text │───▶│ Chunk Text  │───▶│ Embed     │
    │         │    │ (PyPDF)      │    │             │    │ (OpenAI)  │
    └─────────┘    └──────────────┘    └─────────────┘    └───────────┘
                          │
                    ❌ INFORMATION LOST HERE


    COLPALI / VISION RAG PIPELINE:
    ┌─────────┐    ┌──────────────┐    ┌─────────────┐    ┌───────────┐
    │   PDF   │───▶│ Convert to   │───▶│ Embed Images│───▶│ Store in  │
    │         │    │ Images       │    │ (ColPali)   │    │ Vector DB │
    └─────────┘    └──────────────┘    └─────────────┘    └───────────┘
                          │
                    ✓ VISUAL INFO PRESERVED


    AT QUERY TIME:
    ┌─────────┐    ┌──────────────┐    ┌─────────────┐    ┌───────────┐
    │  Query  │───▶│ Embed Query  │───▶│ Find Similar│───▶│ Return    │
    │         │    │ (ColPali)    │    │ Pages       │    │ PAGE IMAGES│
    └─────────┘    └──────────────┘    └─────────────┘    └───────────┘
                                                                │
                                                                ▼
                                                         ┌───────────┐
                                                         │ Vision LLM│
                                                         │ (GPT-4V)  │
                                                         │ "Sees" the│
                                                         │ table!    │
                                                         └───────────┘

Two structural facts fall out of that picture, and both have consequences that outlive the technique.

There is no chunking step. A page is the atom. Everything Part 3 taught you about chunk size, overlap and split boundaries simply evaporates — along with the entire class of bugs it causes. That is a real and under-appreciated benefit: you cannot split a table across two chunks if you never split anything. It is also a real limitation: you get no sub-page granularity, and a question whose answer spans two pages has no mechanism to combine them.

Retrieval hands you a picture, so the generator must have eyes 07:28:43. Choosing ColPali for retrieval forces a vision model for generation. That coupling is where the 10× cost comes from — it is not an implementation detail you can optimise away later.

Against the four losses listed above, the vision path offers four symmetric guarantees [07:25:21–07:25:33]: tables stay as tables, charts stay as charts, layout context is preserved, and there are no text-extraction errors — you cannot garble what you never parsed. That last one is quietly the strongest, because extraction errors are the silent kind: a mis-parsed column produces a well-formed sentence containing a wrong number, and nothing anywhere in your stack throws. The instructor's summation, and the rhetorical core of the chapter 07:26:12:

"That's why multimodal is indeed the king."

And the embedding-symmetry rule from Part 4 comes back harder 07:28:00:

"Remember, we have to use the same model that embedded the documents."

Here the query is text and the documents are images. Only a joint vision-language embedding space makes that comparison meaningful at all. You cannot mix text-embedding-3-small document vectors with ColPali query vectors — and per invariant 2, dimension mismatch is the loud failure. The dangerous one is the silent semantic mismatch where the numbers line up and the results are noise.

What ColPali actually is — and where the course is wrong

The lesson's own summary block:

text
# fcc-production-rag-part-6/06_multimodal_rag.py — lines 142-150
    COLPALI KEY INSIGHT:
    ─────────────────────────────────────────────────────────────────
    ColPali (Contextual Late-interaction for Pali) creates embeddings
    that capture BOTH text AND visual layout in a single vector.

    - Based on PaliGemma (Google's vision-language model)
    - One embedding per document image
    - Query embedding matches against document images
    - No text extraction needed!

Three of those four bullets are fine. The "single vector" claim is wrong, and it is the error most likely to cost you money.

That correction has three consequences you have to plan for, and none of them is in the chapter:

  • Storage. Budget for ~10³ small vectors per page, not one. On a 10,000-page corpus that is millions of vectors, and the per-page footprint is what changes your storage model — not the page count.
  • The store. You need native multi-vector / MaxSim scoring, not merely a column that accepts a float array. Where your store lacks it, the standard fallback is two-stage: keep one pooled vector per page for cheap candidate generation, then rescore a shortlist with MaxSim in your own process. The first stage will sometimes drop a page MaxSim would have surfaced — but that loss is localised to a stage you can measure, which is the difference between an engineering trade-off and a mystery.
  • The reference implementation below. Its mean(dim=1) mean-pooling is that pooled vector — and used alone, with no MaxSim stage after it, it throws away exactly the thing that makes ColPali work.

The reference implementation, read critically

The entire ColPali code in the lesson lives inside a triple-quoted string that is printed, never executed [07:29:56–07:30:19]. There is no colpali_engine, torch, PIL or pdf2image import anywhere in the executable part of the file. Treat what follows as pseudocode that communicates the pipeline shape — the shape is right, the API surface is not. (The listings below drop the # ───── rule lines the file uses as section separators; nothing else is elided.)

Python
# fcc-production-rag-part-6/06_multimodal_rag.py — show_colpali_implementation(), lines 278-314
    # Installation (requires GPU for optimal performance)
    # pip install colpali-engine pdf2image pillow

    from colpali_engine import ColPali
    from pdf2image import convert_from_path
    import torch

    # 1. INITIALIZE COLPALI
    model = ColPali.from_pretrained(
        "vidore/colpali-v1.2",
        torch_dtype=torch.bfloat16,
        device_map="cuda"  # Requires GPU
    )
    processor = model.processor

    # 2. CONVERT PDF TO IMAGES
    def pdf_to_images(pdf_path: str) -> list:
        """Convert each PDF page to an image."""
        images = convert_from_path(pdf_path, dpi=150)
        return images

    # 3. CREATE IMAGE EMBEDDINGS
    def embed_document_images(images: list) -> list:
        """Create ColPali embeddings for each page image."""
        embeddings = []
        for image in images:
            inputs = processor(images=image, return_tensors="pt")
            inputs = {k: v.to(model.device) for k, v in inputs.items()}

            with torch.no_grad():
                embedding = model(**inputs).last_hidden_state.mean(dim=1)

            embeddings.append(embedding.cpu().numpy())
        return embeddings

Every parameter in that block is a decision:

ParameterValueWhy it matters
model idvidore/colpali-v1.2vidore/ is the Hugging Face org (ViDoRe = Visual Document Retrieval). The instructor flags version churn himself: "this could change by the time you watch this" 07:30:06. Pin it, and re-check the docs.
torch_dtypetorch.bfloat16Half precision. Halves VRAM versus fp32 with negligible retrieval-quality loss.
device_map"cuda"Hard-coded GPU. No CPU fallback, no error handling if there is no GPU.
dpi150The only rendering parameter given, and an undiscussed accuracy knob — see the callout below.
torch.no_grad()—Inference only: no autograd graph, far less VRAM. Non-optional at page scale.
pooling.last_hidden_state.mean(dim=1)This is the flaw. Mean-pooling collapses the patch vectors into one and destroys the late interaction. Useful as a first-stage candidate generator; on its own it is not ColPali retrieval.
.cpu().numpy()—Move results off the GPU before storing. Good habit, baked in.

The query side is the mirror image of the document side — same model object, same pooling — which is the symmetry rule made concrete in code. Retrieval is a plain dot product with a default of k=3, and it returns page numbers and PIL images:

Python
# fcc-production-rag-part-6/06_multimodal_rag.py — lines 316-351 (inside the string literal)
    # 4. EMBED QUERY
    def embed_query(query: str):
        """Create ColPali embedding for text query."""
        inputs = processor(text=query, return_tensors="pt")
        inputs = {k: v.to(model.device) for k, v in inputs.items()}

        with torch.no_grad():
            embedding = model(**inputs).last_hidden_state.mean(dim=1)

        return embedding.cpu().numpy()

    # 5. SEARCH AND RETRIEVE
    def search_documents(query: str, document_embeddings: list, images: list, k: int = 3):
        """Find most relevant page images for the query."""
        query_embedding = embed_query(query)

        # Calculate similarities
        similarities = []
        for i, doc_emb in enumerate(document_embeddings):
            sim = np.dot(query_embedding.flatten(), doc_emb.flatten())
            similarities.append((i, sim))

        # Sort by similarity
        similarities.sort(key=lambda x: x[1], reverse=True)

        # Return top-k page images
        results = []
        for idx, score in similarities[:k]:
            results.append({
                "page_number": idx + 1,
                "score": score,
                "image": images[idx]
            })
        return results

Note the free win hiding in "page_number": idx + 1: because the retrieval unit is a page, citation granularity comes for free. "Page 7" is not something you have to reconstruct from chunk metadata — it is the primary key. That is a real advantage over text RAG's page/page_label off-by-one trap from Part 2.

Two defects in that block: np is never imported anywhere in the snippet, and — as established above — a raw unnormalised dot product over mean-pooled vectors is not how ColPali scores anything.

The final stage serialises retrieved PIL images to base64 data URLs and splats them into one message:

answer_with_vision and the end-to-end driver (lines 355–407)
Python
# fcc-production-rag-part-6/06_multimodal_rag.py — inside the string literal
    # 6. ANSWER WITH VISION LLM
    def answer_with_vision(query: str, page_images: list):
        """Send retrieved page images to GPT-4V for analysis."""
        import base64
        from io import BytesIO

        # Convert PIL images to base64
        image_contents = []
        for result in page_images:
            buffered = BytesIO()
            result["image"].save(buffered, format="PNG")
            img_str = base64.b64encode(buffered.getvalue()).decode()

            image_contents.append({
                "type": "image_url",
                "image_url": {"url": f"data:image/png;base64,{img_str}"}
            })

        # Call GPT-4V
        from openai import OpenAI
        client = OpenAI()

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": f"Based on these document pages, answer: {query}"},
                        *image_contents
                    ]
                }
            ]
        )

        return response.choices[0].message.content

    # FULL PIPELINE
    def multimodal_rag_query(pdf_path: str, query: str):
        """Complete multimodal RAG pipeline."""
        # 1. Convert PDF to images
        images = pdf_to_images(pdf_path)

        # 2. Create embeddings (do once, store in vector DB)
        embeddings = embed_document_images(images)

        # 3. Search for relevant pages
        results = search_documents(query, embeddings, images, k=3)

        # 4. Answer with vision LLM
        answer = answer_with_vision(query, results)

        return answer

That comment on step 2 — # 2. Create embeddings (do once, store in vector DB) — is the sentence that makes the cost table legible. Page embedding is amortized; the vision call is not.

Stack warning: this snippet retrieves with colpali-engine + torch but answers through the raw OpenAI SDK, while the executable part of the same file uses langchain_openai.ChatOpenAI. Pick one stack for your own code; mixing them means two auth paths, two retry policies and two places to instrument (Part 7).

The message shape trap

The lesson also contains, commented out, what it calls the "production" call with a real image:

Python
# fcc-production-rag-part-6/06_multimodal_rag.py — demo_vision_llm_analysis(), lines 206-210
    # In production with actual image:
    # response = llm.invoke([
    #     {"type": "text", "text": query},
    #     {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"}}
    # ])

The demo, and its honesty gap

The one live call in the whole lesson asks GPT-4o "Which region is underperforming and by how much?" and gets back 07:29:20:

"The South region is underperforming, with actual sales of $1.5M compared to a target of $1.8M, resulting in a variance of -17%."

And then the "KEY ADVANTAGE" contrast, which is the most useful thing in the chapter [07:29:27–07:29:55]:

  • With text extraction you have: "South $1.8M $1.5M -17%" — a string that "doesn't have a lot of context" and "would give your large language model quite a hard time."
  • With vision the model can see: the red X indicating failure; the context of other regions performing well; the visual hierarchy showing South as the outlier; any trend lines or charts supporting the analysis.

The generalizable point: the visual channel carries judgement, not just data. A red X is a human analyst's verdict encoded in pixels. Text extraction throws away the verdict and keeps the raw number — and then asks a language model to re-derive the judgement from a flattened string.

What is worth lifting from that function verbatim is its system prompt, which is effectively a "how to read a financial page" instruction for a vision model:

Python
# fcc-production-rag-part-6/06_multimodal_rag.py — demo_vision_llm_analysis(), lines 215-226
            (
                "system",
                """You are analyzing a financial document. The user will describe
what's in the document image, and you should answer their question based on
the visual information described.

Focus on:
- Specific numbers and percentages
- Visual indicators (colors, checkmarks, X marks)
- Trends shown in charts
- Relationships between data points""",
            ),

Swap the first paragraph for "You are analyzing the attached document page images" and those four bullets become a reusable VLM reading instruction. They matter because a vision model left to its own devices will describe a page; you want it to interrogate one.

When multimodal RAG earns its cost

The lesson's two lists, with the reason attached to each [07:30:19–07:31:48]:

✅ Perfect forBecause
Financial reportsComplex tables with nested headers, charts showing trends, footnotes and annotations
Technical documentationArchitecture diagrams, flowcharts, code with syntax highlighting
Scientific papersFigures and graphs, mathematical equations, data visualizations
Legal documents"Formatting is important" — formatted contracts, tables of terms, signature pages
Medical recordsDiagnostic images, lab result tables, clinical charts
❌ Not necessary forBecause
Plain text documentsNovels, articles, poems — nothing is lost by flattening
Simple structured dataCSV-like content survives extraction intact
Documents where text extraction already works well"You don't have to go overboard" 07:31:22
Real-time applications"Vision models are slower because there's a lot of processing" 07:31:31
Cost-sensitive applicationsGPT-4o with images costs meaningfully more

Legal gets the strongest wording in the chapter 07:30:51: "most importantly, because formatting is important." In a contract, where a clause sits and what is struck through is legally operative. A flat string loses that, and the loss is not recoverable by a smarter model downstream.

Note that no latency number appears anywhere in the chapter. "Slower" is a qualitative warning only — budget your own measurement (Part 7 gives you the instrumentation to take it).

The cost, and what the numbers are worth

The lesson prints this comparison [07:31:50–07:32:41]:

PipelineEmbeddingQuery
Text RAG~$0.0001 per page~$0.01 per query (GPT-4o-mini)
Multimodal RAG~$0.001 per page (ColPali + GPU)~$0.10 per query (GPT-4o with images)
Ratio10×10×

"~10x more expensive, but necessary for visual content!" 07:32:19

Two things the chapter does not say, both of which change how you should read that table.

First, be honest about the provenance. The instructor hedges it himself — "These are just ballpark numbers to give you an idea" 07:31:50 — and he is right to. None of the four figures is sourced, and the ColPali line is not even the same kind of number as the others: ColPali runs on your own GPU, so "$0.001 per page" is amortized hardware, not an API rate. Keep the direction of the claim (vision RAG costs meaningfully more, roughly an order of magnitude) and treat the specific digits as illustrative. Do not put them in a business case.

Second, do the multiplication the chapter skips. The two lines behave completely differently under load, which is invariant 3 again:

  • $0.001/page at index time is one-time. A 10,000-page corpus costs ~$10 to index, forever.
  • $0.10/query is recurring and scales linearly with traffic. At 10,000 queries/day that is ~$1,000/day against ~$100/day for text RAG — roughly $27,000/month of delta, for the same corpus.

The checklist — and the router the course never builds

The five-item implementation checklist [07:33:46–07:34:32], printed at the end of the lesson:

  1. ☐ GPU for ColPali embeddings — "or you can use cloud GPU, you can buy that."
  2. ☐ pdf2image for PDF → image conversion.
  3. ☐ A vector DB that supports image embeddings — "not all vector DBs support image embeddings."
  4. ☐ GPT-4o or Claude for vision-based answering — "or GPT-5 or 10, depending on when you're watching this video."
  5. ☐ Fallback to text RAG for plain documents.

Four things about that list:

Item 1 is a hard requirement, and the course's own materials contradict each other about it. The checklist says you need a GPU. device_map="cuda" is hard-coded. requirements.txt keeps colpali-engine>=0.3.0 and pdf2image>=1.17.0 commented out with # Uncomment if you have GPU, shipping only pillow>=11.0.0. But the repo README's FAQ claims "Do I need a GPU? No. All code runs on CPU. Part 6's Multimodal RAG section mentions GPU for ColPali, but it's optional." Both statements are technically true only because the ColPali code is never executed anywhere in the course. If you actually run ColPali, you need the GPU.

Item 2 has a missing dependency the course never mentions. pdf2image is a wrapper around poppler (pdftoppm), a system binary. pip install pdf2image does not install it, and the conversion will fail on a clean machine or in a slim container. On Debian/Ubuntu that is apt-get install poppler-utils; add it to your Dockerfile in Part 13's terms.

Item 3 is a procurement constraint, delivered as a passing remark. "Not all vector DBs support image embeddings" 07:34:10 — and once you accept the multi-vector correction above, the requirement is stricter than the chapter implies: you need a store that supports multi-vector late-interaction scoring, not merely one that accepts a float array you happened to derive from an image. That can retroactively rule out the store you chose in Parts 4 and 10 — Chroma, which this course uses throughout, has no MaxSim comparator. The chapter names no compliant database, so this is a question you must answer from vendor docs before you architect; the two-stage fallback above is what you fall back to when the answer is no.

Item 5 is an architecture instruction wearing a nicety's clothes. "Sometimes you may have a mixture of different kinds of documents, and so you have to have that in place" 07:34:26. Most real corpora are mixed. Without a router you have exactly two options, and both are bad: pay the vision price on every plain-text PDF, or lose every table. The chapter shows no code, no heuristic and no classifier for this. It is the largest unfilled gap in the multimodal material, so here is the shape it should take:

Python
# Text-vs-vision routing for a mixed corpus — not from the course repo.
# Runs at INDEX time (per document), not per query: the decision is a property
# of the document, and re-deciding it per request is pure recurring cost.
# Needs PyMuPDF (`pip install pymupdf`); `find_tables` requires >= 1.23.
import pymupdf  # historically imported as `fitz`; both names still work


def route_document(pdf_path: str) -> str:
    """Return 'vision' or 'text' for one PDF. Cheap, deterministic, auditable."""
    chars = images = tables = 0
    with pymupdf.open(pdf_path) as doc:
        pages = max(doc.page_count, 1)
        for page in doc:
            chars += len(page.get_text("text"))
            images += len(page.get_images(full=True))
            tables += len(page.find_tables().tables)

    # 1. Almost no extractable text => scanned document. Vision, no debate.
    if chars / pages < 200:
        return "vision"
    # 2. Any real table or figure density => layout carries meaning.
    if (tables + images) / pages >= 1.0:
        return "vision"
    # 3. Otherwise the meaning survives flattening.
    return "text"

Route once at ingest, store the decision in the document's metadata, and index each document into the path it was routed to. Then at query time you either search both indexes and fuse (Part 6's RRF applies unchanged), or you route the query by which index its filters point at. The thresholds above are starting points to calibrate against your own corpus, not constants — but a crude router that is measured beats an elegant one that is assumed.

The lesson's own six takeaways [07:32:43–07:33:43], printed at the end of the file:

1. Text extraction loses tables, charts and diagrams. 2. ColPali embeds document IMAGES, not extracted text. 3. Vision LLMs (GPT-4o, Claude) can "see" the documents. 4. Best for financial reports, technical docs, anything visual. 5. Trade-off: ~10× more expensive, but preserves information. 6. "The future: documents are images, not text."

He offers the sixth as a discussion prompt, not a fact — "that's something for you to think about, and perhaps start discussing" 07:33:39 — and that framing is the right one. The claim is directionally interesting and completely unevidenced in the source. Treat it as a hypothesis you can test on your own corpus with an eval set, which is more than the chapter does.

3 · Where this is going

The last three chapters of the source are a recap, a timeline and a single word. Together they take about three and a half minutes and contain no new code — but they are the chapters that tell you what to actually do, so treat every line as load-bearing.

The six advanced techniques, as one decision table

This is the Part-6 summary table the instructor reads at [07:34:48–07:36:50], with the numbers from the chapters where they were actually established. Note the structure of his own thinking: every row answers "when do I use this?" Nothing is ranked "best."

#TechniqueSlide captionWhat it doesWhen to use itEvidence (from its own chapter)
1Long context vs RAGCOST OPTIMIZATIONChoose the right approachRAG for cost + scale; long context for small, static sets~10¢/query vs near-zero; ~1 s vs ~45 s latency; 25× on the course's own worked example (the spoken "over 1,200×" is never derived — Part 14); long-context reliability degrades around 60–70% of the stated limit
2Contextual retrievalDYNAMIC CONTEXT WINDOWSAdd LLM-written context to each chunk before embeddingAlways — the only unconditional row49% fewer top-20 retrieval failures (contextual embeddings plus contextual BM25, which the course mislabels as "alone"), 67% once reranking is added; 1¢–5¢ per document, one-time (Part 14)
3Late chunkingBETTER COHERENCE & FLOWEmbed first, chunk laterDocuments with references (anaphora, not bibliographies)+10–12% accuracy; contextual chunks grow 20–30%; combined approaches 25–30% (Part 14)
4Agentic RAGAUTONOMOUS MULTI-STEP REASONINGSelf-correcting retrievalProduction systemsGrade → rewrite → retry loop on LangGraph; max_retries typically 2–5 (Part 15)
5GraphRAGSTRUCTURED KNOWLEDGE EXPLORATIONKnowledge-graph reasoning: entities + relationships, traversedMulti-hop questionsSkip below 100 documents; community detection worth it around 100 entities (this part)
6Multimodal / ColPaliIMAGE + TEXT + AUDIO INTEGRATIONVision-based retrievalDocuments with tables, charts, images"Very powerful but expensive as you saw" — ~10× (this part)

The slide's own banner is "The 2026 Production RAG Stack" — stack, not menu. That word choice is the argument, and the checklist below enforces it.

The "static" in row 1 is doing quiet work and is easy to skim past: the discriminator is not only corpus size, it is churn. A small corpus that changes constantly still leans long-context, because you avoid re-embedding. And row 1 is not an either/or: the compressed phrasing "RAG for cost and scale" reads like "never use long context," which is not what Part 14 argues. Pick per workload; the two coexist in one product.

Row 2 is the only unconditional recommendation in the entire course. Asked when to use contextual retrieval, the answer is "Well, always", and the justification is one beat long 07:34:48:

"Why? Because 67% fewer failures. That is worth the trouble if I may say."

"Worth the trouble" is the operative judgement, and it concedes there is trouble: one LLM call per chunk at index time, 20–30% larger chunks, more storage, more indexing latency. The ruling is that a one-time indexing cost dominates a recurring failure rate. Two caveats the recap drops and you should not: the 67% is Anthropic's benchmark, not yours, and it is a stack number, not a technique number — Part 14 unpacks it as contextual embeddings plus contextual BM25 reaching 49%, with reranking supplying the rest. Strip the hybrid search or the reranker out and you are not measuring against 67% any more. Part 14's chapters also repeat "you have to test on your own." Do not let a bare "67%" harden into a guarantee.

Four eras, and the fact that none of them ended

The closing slide's timeline 07:37:02:

EraNameWhat it added
2023Naive RAGchunk → embed → retrieve → generate. One shot, no scoring, no feedback, no fallback.
2024Optimized RAGHybrid search (dense + BM25 fused with RRF), reranking, "and so forth." Same pipeline shape, better retrieval quality.
2025Intelligent RAGContextual retrieval and self-correcting behaviour. The first era where the retriever stops being a dumb function.
2026 and forwardAgentic RAGAutonomous, multimodal, graph-enhanced. Control flow moves out of your code and into the agent.

The eras are additive, not sequential replacements, and the checklist below proves it: reranking is a 2024 technique sitting at #2 on the 2026 list. Dropping hybrid search and reranking because you added an agent is a regression wearing a modernization's clothes.

The four-item production checklist

The payload of the whole closing sequence [07:37:02–07:38:03]. The ordering is the instructor's, and it encodes priority:

  1. Contextual retrieval — "at minimum." Not an optimization; the entry ticket. "You at least need contextual retrieval at minimum." Consistent with the recap's "when to use it? Well, always."
  2. Reranking. Second-pass relevance scoring over retrieved candidates.
  3. Agentic patterns — "this is the core." An agent that evaluates what came back, decides whether it is good enough, reformulates and retries. The argument for it is precise 07:37:02:

"A robust system that doesn't depend on you or just your code that you put in production."

With a hard-coded pipeline, every failure mode you did not anticipate before deploy is a permanent bug. With an agent, retrieval decisions are made at runtime, by something that can look at what came back and react. Note the scope, because it is easy to over-read: the decisions he means are "critical decisions when it comes to retrieval" 07:37:02 — is this candidate set good enough, should I rewrite the query, should I retry — not free-roaming autonomy over your product. And within the agentic bucket he singles out one capability 07:38:03: "self-correcting, which is the most important part."

  1. Multimodal support. The justification is data-driven, not capability-driven: your corpus already contains images, charts and tables that your text-only pipeline is silently dropping at ingest. The slide goes further than the narration here — its card reads "IMAGE + TEXT + AUDIO INTEGRATION." Don't narrow "multimodal" to PDFs-as-images.

There is an implicit step 0 hiding in the timeline: you are assumed to already have hybrid search underneath. Reranking with nothing good to rerank is theatre.

The list is pitched as durable rather than versioned — "this year and next year and years to come" 07:37:02 — which is the right register. Nothing in those four items names a model, a library version or a vendor. That is what makes it survivable: every specific in this guide will rot, and this ordering will not.

Notice what is not on that list, and hold the omission against the source: evaluation. There is no "measure your retrieval quality" item, no eval-set item, no regression-test item — despite the fact that every recommendation above is gated on a metric ("start with recursive, measure your retrieval quality"; "test on your own"). A four-item checklist that tells you to add contextual retrieval, reranking, an agent and vision, but not to measure whether any of them helped, is describing a system that cannot tell you when it has regressed. That is Part 7's and Part 15's job and it belongs on this list.

The verdict, and the honest state of the frontier

The course's closing couplet is delivered across a chapter boundary. The final chapter of 32 is literally one word — RAG., three seconds at 07:38:37 — because the YouTube marker at 07:38:35 lands in the middle of the last sentence. There is no end card, no recap, no next steps, no repo URL; the video hard-cuts at 07:38:38, its full runtime. Which means the source supplies no conclusion to inherit, and the closing judgement is the only thing it hands you:

"So the naive chunk-and-pray strategy is dead. Well, welcome to the future of RAG." [07:38:31–07:38:39]

"Chunk and pray" is the right pejorative for the right thing: split on a fixed size, embed, take top-k, hope the right chunk shows up, ship it, and have no instrumentation to know when it doesn't. That is no longer a defensible production design, and this guide has spent fifteen parts on why.

But be clear-eyed about which parts of "the future" this material actually delivers, because the gap is wide and knowing it is the difference between building and cargo-culting:

ClaimStatus in the sourceWhat you must supply
GraphRAG answers multi-hop queriesDemonstrated — on a hand-authored 6-node graph with a hardcoded traversalReal extraction, entity resolution, graph persistence, NL→graph-pattern query parsing
GraphRAG automates graph construction with an LLMHalf-demonstrated — the extractor runs, but nothing parses its output back into nodes and edgesStructured output, a loader, dedup
Local vs global searchExplained, never implementedA router, or expose the mode as a parameter
ColPali preserves what extraction destroysAsserted, never demonstrated — the one live call sends text, not an imageAn actual end-to-end ColPali run on your corpus, before you budget for it
ColPali gives "one embedding per image"Wrong — it is multi-vector late interaction (a grid of patch vectors per page, scored with MaxSim)A store that can do multi-vector scoring, or a two-stage pooled-then-MaxSim retriever
What "late interaction" meansNamed in the acronym, never explained — no ColBERT, no MaxSim, no multi-vector discussion anywhereThe mechanism, before you size storage or pick a store
Multimodal costs ~10×Ballpark, unsourced, by the instructor's own admissionYour own measurement; the direction is right, the digits are not
"Fallback to text RAG for plain documents"Named as a checklist item, no codeThe router (a sketch is above)
Vision RAG accuracy vs text RAGNo benchmark anywhere — no ViDoRe numbers, no deltaYour own eval set
Latency of vision embedding / answering"Slower." No number.Your own measurement
GPU sizing behind "$0.001 per page"Never derived — no card, no throughput, no $/hourA benchmark on the GPU you can actually rent
PaliGemma licence and hosting termsNot mentioned at allA licence review before a commercial deployment
Environment and version pins for the Part 6 code (both files in this part)

The two lesson files live in the second repo, fcc-production-rag-part-6 — the one you bootstrap separately in Part 2. Its declared environment:

  • pyproject.toml: requires-python = ">=3.13", with .python-version pinned to 3.13. The course was visibly running Python 3.12.2 on screen (VS Code status bar throughout chapter 29), and the repo README claims 3.10+. Three different answers; trust the code you can run.
  • The repo carries two dependency files that disagree, which is itself the lesson: requirements.txt is the wide one (langchain>=1.2.0,<2.0.0, langchain-core>=1.3.0,<2.0.0, langchain-openai>=1.1.0,<2.0.0, langchain-chroma>=1.1.0,<2.0.0 — not langchain-community, per an explicit comment in the file — chromadb>=1.5.0,<2.0.0, langgraph>=1.1.0,<2.0.0, langchain-text-splitters>=1.0.0,<2.0.0, tiktoken>=0.12.0, pypdf>=6.0.0, python-dotenv>=1.0.0); pyproject.toml lists a narrower eight-package subset and adds openai>=2.32.0, which requirements.txt never names. Install from one of them, not both.
  • GraphRAG (lesson 6.5): networkx>=3.3 in requirements.txt, networkx>=3.6.1 in pyproject.toml, plus pyvis>=0.3.2 — pinned for graph visualisation, in requirements.txt only, and never imported by any file.
  • Multimodal (lesson 6.6): only pillow>=11.0.0 is installed. colpali-engine>=0.3.0 and pdf2image>=1.17.0 are commented out with # Uncomment if you have GPU. Neither is imported by the executable part of the file, which is why the whole 526-line lesson runs in 2 seconds and makes exactly one API call.
  • Also present: sentence-transformers>=5.0.0 (reranking, optional), langchain-classic>=1.0.0 (legacy retrievers), langchain-anthropic>=0.4.0 (contextual retrieval, Part 14).

Model ids across the six Part 6 files: gpt-4o-mini in lessons 1, 2, 4 and 5; gpt-4o in lesson 6; text-embedding-3-small in lessons 1–4. Lesson 1 also references gpt-5.4-nano in its cost arithmetic. No file in the repo has a test, a type check, an API-level retry, a backoff, a timeout or a token-budget guard — every LLM call is a bare chain.invoke(...). The single try/except in all six files (lesson 4, line 191) guards a float() parse of a grader's output, not a network call; lesson 4's max_retries bounds the retrieval loop, not API failures. That is the distance between this course's code and the word "production" in its title, and Parts 11–13 of this guide exist to close it.

None of that makes the techniques wrong. It makes them frontier — which is exactly what this part is called. The honest posture for an infrastructure engineer is: the mechanisms are real and the failure modes they address are real, the demonstrations are thin, and the numbers are directional. Build the eval harness first, then adopt one technique at a time and keep the one that moves the number.

Decisions to make

Work through these in order. Each is a fork you should resolve deliberately rather than by default.

Do you have a multi-hop problem at all?

  1. Pull 100 real queries from your logs. Label each: does answering it require chaining facts from two or more documents? If fewer than ~10% are multi-hop, stop — you do not need a graph, and Parts 6 and 14 will buy you more.
  2. Check the failure signature before you build anything: were the right chunks in context and the answer still wrong? That is a missing join, not missing recall. If the right chunks were not in context, fix retrieval first.
  3. Check scale: under 100 documents, skip GraphRAG. Under 50,000 tokens total, consider long context instead (Part 14).
  4. Check churn: if your corpus needs near-real-time indexing, GraphRAG is disqualified. Extraction plus community re-summarisation is not an append-friendly operation.

If you are building the graph:

  1. Choose your implementation: Microsoft GraphRAG for a full, opinionated implementation (heavy indexing compute); Neo4j + GraphCypherQAChain if you already run Neo4j and want control (read-only credentials, allow_dangerous_requests=True understood as the warning it is); LlamaIndex PropertyGraphIndex for the easy path; hybrid vector + graph if you want to keep your existing retriever and add traversal on top.
  2. Design your relation vocabulary before you extract, in SCREAMING_SNAKE_CASE, and pin it in the extraction prompt. Traversal matches relation strings exactly.
  3. Use structured output (Pydantic) for extraction, not free-text formatting instructions. You need objects, not prose.
  4. Budget explicitly for entity resolution. It is not optional and the source never mentions it.
  5. Decide how you persist the graph and how you update it incrementally. NetworkX in memory is a demo.
  6. Return the reasoning path with every graph answer. It is your citation.
  7. Keep vector search alongside the graph. Step 4 is hybrid; a graph-only retriever loses recall on every non-relational query.

Do you have a vision problem?

  1. Open ten random documents from your corpus and run your current extractor over them. If the tables and figures survive as usable text, stop — you don't need this.
  2. Classify by document type: financial / technical / scientific / legal / medical → probably yes. Novels, articles, CSV-like → no.
  3. Multiply before you commit: queries/day × $0.10 versus queries/day × $0.01. At 10k queries/day the delta is roughly $27k/month. Is the corpus worth it?
  4. Check your latency budget. Vision models are slower and no one has told you by how much.
  5. Prove the premise before you fund it. Take twenty pages your extractor mangles, run a real ColPali retrieval and a real vision answer over them, and compare against your text pipeline on the same questions. Nobody in the source chain has done this; the whole thesis is untested until you test it.

If you are building the vision path:

  1. Secure a GPU — local or cloud. The GPU is not optional, whatever the README says.
  2. Install poppler (poppler-utils) alongside pdf2image. It is a system package.
  3. Verify your vector store supports multi-vector late-interaction scoring, not merely "image embeddings." Do this before you architect. If it does not, decide now between switching stores and building the two-stage pooled-then-MaxSim retriever — and size storage for ~10³ vectors per page either way.
  4. Pin vidore/colpali-v1.2 and set a reminder to re-check the model card. Model ids rot.
  5. Calibrate dpi. 150 is the default; test 200–300 on your densest pages and read what the model gets wrong.
  6. Never let a mean-pooled vector be your final score. Pooling is a legitimate first stage; ending there means you bought a vision pipeline and threw away its retrieval quality.
  7. Build the text-vs-vision router at index time, store the decision in metadata, and keep both paths alive.
  8. Reuse the four-bullet VLM reading instruction (numbers, visual indicators, chart trends, relationships between data points) as your system prompt. A vision model left alone will describe the page instead of interrogating it.
  9. Lean on the free citation granularity: page images give you "page 7" as a primary key, not as a reconstruction.

Before you call any of it done:

  1. Run the 2026 checklist against your system, in order: contextual retrieval (the floor) → reranking → agentic patterns (the core, and self-correction inside it) → multimodal support. Confirm nothing from an earlier era was dropped on the way — hybrid search and reranking survive the arrival of an agent.
  2. Add the item the source's checklist omits: an eval set and a regression gate. Every technique in this part is a "test on your own" claim. Without a measurement you cannot tell adoption from decoration — and per invariant 4, nothing will throw an exception to tell you.