oriolrius.me Production RAGReference guide
Part 06·Quality & operations

Retrieval Quality: Hybrid Search, Reranking and Citations

Hybrid BM25 + vector search fused with RRF, reranking, compression and citations.

45 min read 28 sections Quality & operations
Audio summary · 5:54 · EN
Retrieval Quality: Hybrid Search, Reranking and Citations

Parts 3 to 5 got you a system that retrieves something and answers with it. This part is about the gap between something and the right thing. Every technique here modifies exactly two boxes of the Phase B pipeline from Part 1 — Retrieve → Fuse/Filter → Compress/Rerank — and none of them touches the index. That is deliberate: the cheapest quality upgrades you can ship are the ones that do not force a re-embed.

Two of the five failure modes from Part 1 live here:

  • Failure mode #2, embedding mismatch. The user types SKU-7742X or E_CONN_REFUSED and the embedding model, which only knows meaning, has no meaning to work with. Fix: hybrid search — add a retriever that does not care about meaning at all.
  • Failure mode #3, retrieval noise. Ten documents come back, two are relevant, and the eight distractors eat your context window and your model's attention. Fixes: reranking, compression, and parent-document retrieval.

Both failures are silent (invariant 4). Nothing raises. Your error rate stays at 0% while your answers get quietly worse, which is why every technique in this part ships with a measurement harness rather than a vibe.

Compressed to one sentence, this part says: retrieve wide, then narrow deliberately. Every technique below is a different way of widening or of narrowing, and the skill is knowing which one a given failure calls for.

Where pure vector search collapses

Vector search's superpower is that it matches meaning instead of characters. That is also precisely its blind spot: strings that carry no meaning have no position in semantic space 01:53:47. The instructor is careful to frame these as total failures, not degradations — the correct document sits untouched in the database while the retriever confidently returns four wrong ones.

There are four classes, and they are worth memorising because they are how you will recognise the disease in your own logs.

The four canonical breakages, each with the wrong documents vector search actually returns. Note the pattern in the distractors: they are all topically adjacent and none of them contain the literal string. That is the signature — plausible neighbours, zero exact matches.
Figure 1The four canonical breakages, each with the wrong documents vector search actually returns. Note the pattern in the distractors: they are all topically adjacent and none of them contain the literal string. That is the signature — plausible neighbours, zero exact matches.
Failure classExample queryWhat vector search returns insteadRoot cause
Product codes / SKUsSKU-7742X specsToaster X-500, Tool-88, Per-98, Shoe-99No semantic meaning — just characters
Error codes / identifiersE_CONN_REFUSEDDatabase Manual, Network Setup Guide, General TroubleshootingJust characters to the embedding model
Acronyms / abbreviationsWCAG complianceWorkout Plan, Water Conservation Standards, Wireless StandardsThe model doesn't know the abbreviation expands to Web Content Accessibility Guidelines
Exact namesJohn Smith accountingSmith Family Tree, Accounting Principles, John's BakerySemantics override specifics

Look at the acronym row closely, because it is the most instructive. The retriever did not return random noise. It returned documents whose initials or vibe line up — Workout, Water Conservation, Wireless. The embedding encoded "something W-ish and standards-y" and found the nearest such things. It behaved perfectly. It just had nothing to behave perfectly on.

The exact-names row has the crispest diagnosis on the whole slide: semantics override specifics. The query "John Smith accounting" decomposes, in embedding space, into a "person" direction and an "accounting" direction. Documents that are strongly one or the other beat the document that is weakly both. The specific entity is drowned by the general topic.

Two related-but-different failures you should not confuse with these, because their fixes live elsewhere:

  • The chunk was never coherent to begin with (the answer is split across chunk 47 and chunk 48) — that is Part 3, and no retriever fixes it.
  • The bridging entity's name isn't in the query at all ("the CEO's assistant" vs "Sarah Johnson works in the Executive department") — no amount of BM25, reranking or k-tuning helps, because neither branch has a string or a concept to latch onto. That is the multi-hop problem, and it is Part 16.

BM25 ("Best Matching 25") is a lexical ranking function: it scores a document by the statistics of the query terms inside it — term frequency, document length normalisation, and inverse document frequency so that rare terms count for more than common ones. No embedding model, no neural network, no semantics.

"BM25 is the opposite of vector search. It doesn't understand meaning at all. It just counts words." 01:56:52

That sounds like a limitation until you line the two up against each other.

The two columns are near-perfect mirror images: every item in one engine's GOOD list appears in the other's BAD list. That symmetry is why fusion works — the two retrievers make uncorrelated errors, which is the precondition for any ensemble to beat its members.
Figure 2The two columns are near-perfect mirror images: every item in one engine's GOOD list appears in the other's BAD list. That symmetry is why fusion works — the two retrievers make uncorrelated errors, which is the precondition for any ensemble to beat its members.
Vector searchBM25 search
Good atsemantic similarity · synonyms · natural questionsexact matches · rare terms · codes & IDs
Example"How do I authenticate?" → finds 'login credentials'SKU-7742X → finds the exact match
Bad atexact matches · product codes · acronymssynonyms · semantic meaning
Needs an embedding modelYes, at index and query timeNo
Index update costIncremental (upsert)Full rebuild
Where it runsVector DBIn-process (rank_bm25) or in your search engine

The first three rows are the slide's; the last three are what the shipped code adds, and they are the ones that decide your architecture. The line the whole lesson turns on is the slide's own: "What one MISSES, the other CATCHES", which it resolves into "HYBRID = BEST OF BOTH" 01:57:53. Take the first half as the engineering claim and the second half as the marketing — the measured results later in this part support "rarely worse, sometimes much better", which is a weaker and more useful statement than "best of both".

What BM25Retriever actually does

Do not treat BM25 as a black box. The LangChain wrapper is barely a hundred lines, it is installed in the pinned environment, and two of its defaults will bite you.

Python
# .venv/.../langchain_community/retrievers/bm25.py — the shipped implementation (abridged)
def default_preprocessing_func(text: str) -> List[str]:
    return text.split()


class BM25Retriever(BaseRetriever):
    """`BM25` retriever without Elasticsearch."""

    vectorizer: Any = None
    docs: List[Document] = Field(repr=False)
    k: int = 4
    """ Number of documents to return."""
    preprocess_func: Callable[[str], List[str]] = default_preprocessing_func

    @classmethod
    def from_texts(cls, texts, metadatas=None, ids=None, bm25_params=None,
                   preprocess_func=default_preprocessing_func, **kwargs):
        from rank_bm25 import BM25Okapi

        texts_processed = [preprocess_func(t) for t in texts]
        vectorizer = BM25Okapi(texts_processed, **(bm25_params or {}))
        ...
        return cls(vectorizer=vectorizer, docs=docs,
                   preprocess_func=preprocess_func, **kwargs)

    def _get_relevant_documents(self, query, *, run_manager):
        processed_query = self.preprocess_func(query)
        return self.vectorizer.get_top_n(processed_query, self.docs, n=self.k)

Four things fall out of those lines:

  1. The default tokenizer is str.split(). Whitespace only. No lowercasing, no punctuation stripping, no stemming. So E_CONN_REFUSED. at the end of a sentence is a different token from E_CONN_REFUSED, and WCAG does not match wcag. A BM25 branch that silently underperforms on exactly the codes you added it for is usually this, and nothing else.
  2. Pass preprocess_func at construction, never afterwards. Look at where it is used: from_texts runs it over the whole corpus before building the vectorizer, and _get_relevant_documents runs it over the query. Assigning retriever.preprocess_func = my_func after the fact changes only the query side, so query and index are tokenized by different rules and your recall quietly collapses. Pass it as a keyword to from_documents(...) so both sides agree.
  3. k defaults to 4, not 3 — which happens to match the production advice on slide 06. The demo sets bm25_retriever.k = 3 on the instance afterwards; that works, but so does BM25Retriever.from_documents(docs, k=3), because from_documents forwards **kwargs through from_texts into the constructor. Prefer the keyword: it keeps the retriever's configuration in one expression.
  4. docs is the entire corpus, held in memory, and get_top_n scores every document on every query. rank_bm25's BM25Okapi is instantiated with k1=1.5, b=0.75, epsilon=0.25 and does a linear scan with np.argsort over all scores — fine at thousands of documents, a real problem at millions. At that scale you move the keyword branch into Postgres full-text search, OpenSearch/Elasticsearch, or your vector DB's native hybrid mode (Part 10).

And the mandatory warning in the other direction: never ship BM25 alone. In the live walkthrough, "How do I authenticate?" put the WCAG 2.1 compliance document at rank #1 — "totally failed" 02:09:05. Even on queries BM25 got right at rank 1, ranks 2 and 3 were unrelated documents 02:07:04. Top-1 precision does not imply a clean list.

Split then fuse: the hybrid pipeline

Hybrid search is not "vector search with a keyword filter". The query goes down both branches simultaneously and the two ranked lists are merged by a fusion algorithm 01:58:56.

Figure 3Hybrid search: vector and BM25 rankings merged by Reciprocal Rank Fusion
Original ASCII
                    ┌──────────────────────┐
                    │  "SKU-7742X specs"   │
                    └──────────┬───────────┘
                               │  (the same query string, unmodified)
              ┌────────────────┴────────────────┐
              ▼                                 ▼
      ┌───────────────┐                 ┌───────────────┐
      │ VECTOR branch │                 │  BM25 branch  │
      │ embed → ANN   │                 │  term stats   │
      └───────┬───────┘                 └───────┬───────┘
              │ doc3, doc7, doc1                │ doc1, doc9, doc3
              └────────────────┬────────────────┘
                               ▼
                 ┌──────────────────────────┐
                 │  Reciprocal Rank Fusion  │   weights=[w_bm25, w_vector]
                 │  score += w / (rank + c) │   c = 60
                 └────────────┬─────────────┘
                              ▼
                     doc1, doc3, doc7  ← one fused ranking

Note what does not happen: the query is not rewritten, split or routed. The same string goes to both branches. That matters because it means hybrid search costs you no LLM call — unlike multi-query later in this part, which does exactly the opposite trade.

RRF, and why consensus wins

Reciprocal Rank Fusion throws away the scores. That is the point: a cosine distance of 0.31 and a BM25 score of 14.2 are not on the same scale and never will be, so RRF uses the only thing the two lists share — position. Here is the shipped implementation, which is worth reading because you will occasionally need to re-derive its behaviour from first principles when the fused output surprises you.

Python
# .venv/.../langchain_classic/retrievers/ensemble.py — EnsembleRetriever (abridged)
class EnsembleRetriever(BaseRetriever):
    retrievers: list[RetrieverLike]
    weights: list[float]
    c: int = 60
    id_key: str | None = None

    @model_validator(mode="before")
    @classmethod
    def _set_weights(cls, values: dict[str, Any]) -> Any:
        if not values.get("weights"):
            n_retrievers = len(values["retrievers"])
            values["weights"] = [1 / n_retrievers] * n_retrievers
        return values

    def weighted_reciprocal_rank(
        self, doc_lists: list[list[Document]]
    ) -> list[Document]:
        if len(doc_lists) != len(self.weights):
            msg = "Number of rank lists must be equal to the number of weights."
            raise ValueError(msg)

        rrf_score: dict[str, float] = defaultdict(float)
        for doc_list, weight in zip(doc_lists, self.weights, strict=False):
            for rank, doc in enumerate(doc_list, start=1):
                rrf_score[
                    doc.page_content
                    if self.id_key is None
                    else doc.metadata[self.id_key]
                ] += weight / (rank + self.c)

        all_docs = chain.from_iterable(doc_lists)
        # `_key` below stands for the same page_content-or-id_key ternary, which the
        # source inlines a third and fourth time here.
        return sorted(
            unique_by_key(all_docs, _key),
            reverse=True,
            key=lambda doc: rrf_score[_key(doc)],
        )

Two things to bank before the arithmetic. First, _set_weights: omitting weights entirely does not raise — it silently gives every retriever 1/n. That is a fine default and a terrible surprise if you thought you had configured a lean. Second, the constant c = 60 is the damping term, and it is the reason the algorithm behaves the way it does. With c = 60, rank 1 contributes w/61 and rank 3 contributes w/63 — a difference of about 3%. Ranks are treated as nearly equally good, so being present in both lists matters far more than being first in one.

Work the arithmetic on the 50/50 case:

DocumentBM25 rankVector rankRRF scoreFused position
doc A1absent0.5/61 = 0.00820tied 2nd
doc B330.5/63 + 0.5/63 = 0.015871st
doc Cabsent10.5/61 = 0.00820tied 2nd

A document that is #3 in both lists beats a document that is #1 in one and missing from the other 01:59:58. That is the emergent property people mean when they say hybrid "naturally balances semantic relevance with keyword precision." It is not magic; it is c = 60 making agreement cheap and disagreement expensive. (Ties break by whatever order sorted encountered them in — sorted is stable, and chain.from_iterable walks the retriever lists in declaration order, so on an exact tie the first retriever in your list wins. One more reason the retriever order is not cosmetic.)

One subtle trap hiding in that code: the deduplication key defaults to doc.page_content — the entire text. Two chunks with identical text from different source files collapse into one and their scores add. Conversely, if you ran the chunks through contextual retrieval (Part 14) so each carries a slightly different LLM-written preamble, two copies of the same chunk will not dedupe. Set id_key to a stable document-id field in metadata whenever your corpus has real identifiers — and note that id_key is read as doc.metadata[id_key], a hard subscript, so a single document missing that field raises KeyError mid-fusion.

Building it: three components, three lines

The build is genuinely small 02:01:59: a vector retriever you already have ("the core of semantic understanding"), a BM25 retriever that is one line, and an ensemble that fuses them. Install the keyword engine first 02:02:59:

bash
uv add rank-bm25

The versions this part was verified against, straight out of the repo's pyproject.toml:

PackagePin
python>=3.12
langchain>=1.2.7
langchain-openai>=1.1.7
langchain-chroma>=1.1.0
langchain-community>=0.4.1
rank-bm25>=0.2.2
langgraph>=1.0.7

Now the corpus, and here you have to separate two things the source blurs. The video demonstrates hybrid search on a deliberately adversarial corpus that mixes SKUs, error codes, OAuth prose, router configuration and IP addresses, "to emulate a real use case where you have documents of this nature" 02:02:59 — that is the corpus behind the five-query results table below, and its file is not in either published repo. The committed demo runs on TECH_DOCS, eight clean paragraphs about Python, JavaScript, ML, LangChain, LangGraph, Docker, PostgreSQL and vector databases, with no SKU or error code anywhere in it.

Build yours like the video's, not like the repo's. A corpus of clean prose will tell you hybrid search does nothing, and it will be lying to you — this is the same trap that makes the compression demo look broken later in this part.

Python
# production-course-main-code/advanced_rag.py — demo_ensemble_hybrid_search() (abridged)
from langchain_classic.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever


def demo_ensemble_hybrid_search():
    """Hybrid search combining keyword (BM25) and semantic search."""

    vectorstore = create_base_vectorstore()

    # BM25 keyword retriever
    bm25_retriever = BM25Retriever.from_documents(TECH_DOCS)
    bm25_retriever.k = 3

    # Semantic retriever
    semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

    # Ensemble combines both
    ensemble_retriever = EnsembleRetriever(
        retrievers=[bm25_retriever, semantic_retriever],
        weights=[0.4, 0.6],  # 40% keyword, 60% semantic
    )

    queries = [
        "ACID transactions",                                    # Keyword-heavy (BM25 helps)
        "How do I store AI model outputs for later retrieval?", # Semantic (vectors help)
        "fast similarity lookup for embeddings",                # Mixed
    ]

    for query in queries:
        bm25_results = bm25_retriever.invoke(query)
        semantic_results = semantic_retriever.invoke(query)
        ensemble_results = ensemble_retriever.invoke(query)

        print(f"BM25 top result: {bm25_results[0].page_content[:60]}...")
        print(f"Semantic top result: {semantic_results[0].page_content[:60]}...")
        print(f"Ensemble top result: {ensemble_results[0].page_content[:60]}...")

Notice the discipline in that query list: every test query is annotated with the category it probes. Keyword-heavy, semantic, mixed. That comment is not decoration — it is the only way to read the output and know which retriever should have won, which is the prerequisite for tuning weights on evidence instead of taste.

The dependency-free fallback

Because the import path is unstable and because you may want fusion outside LangChain entirely, keep a 20-line RRF in your own codebase. This is a faithful reimplementation of langchain_classic/retrievers/ensemble.py::weighted_reciprocal_rank — same constant, same accumulation, same dedup rule, no framework dependency.

Hand-rolled weighted RRF (dependency-free)
Python
# Reimplementation of langchain_classic/retrievers/ensemble.py::weighted_reciprocal_rank
# Same algorithm, same c=60. Use when the import path moves or you are not on LangChain.
from collections import defaultdict
from itertools import chain


def weighted_rrf(doc_lists, weights, c=60, id_key=None):
    """Fuse N ranked lists of Documents into one ranking.

    doc_lists: list of ranked lists, best-first.
    weights:   one float per list, positionally matched.
    c:         rank damping constant; larger => ranks matter less, consensus more.
    id_key:    metadata field to dedupe on; falls back to page_content.
    """
    if len(doc_lists) != len(weights):
        raise ValueError("Number of rank lists must equal the number of weights.")

    def key(doc):
        return doc.page_content if id_key is None else doc.metadata[id_key]

    scores = defaultdict(float)
    for doc_list, weight in zip(doc_lists, weights):
        for rank, doc in enumerate(doc_list, start=1):
            scores[key(doc)] += weight / (rank + c)

    seen, unique = set(), []
    for doc in chain.from_iterable(doc_lists):
        k = key(doc)
        if k not in seen:
            seen.add(k)
            unique.append(doc)

    return sorted(unique, key=lambda d: scores[key(d)], reverse=True)


# Usage — the ensemble is a LIST, not a pair. N retrievers, N weights.
fused = weighted_rrf(
    [bm25_retriever.invoke(q), vector_retriever.invoke(q)],
    weights=[0.5, 0.5],
)

The ensemble generalises to N branches, not just two:

"If we had 10 retrievers, this is a list. We could have put all of them there." 02:05:03

A third branch — a metadata-filtered retriever, a second embedding model, a graph retriever — costs one more entry in retrievers and one more float in weights. The only constraint the code enforces is that the two lists are the same length; everything else is your judgement.

Tuning the weights

weights is a slider between two poles, and it maps positionally onto retrievers.

The player timer pins this slide to 02:06:14. The starred midpoint is the only defensible starting position: you do not yet know your query mix, and 50/50 is the setting that is wrong by the smallest margin in either direction.
Figure 4The player timer pins this slide to 02:06:14. The starred midpoint is the only defensible starting position: you do not yet know your query mix, and 50/50 is the setting that is wrong by the smallest margin in either direction.
Weights [bm25, vector]LeansUse when
[0.7, 0.3]More BM25Lots of product codes, error codes, IDs
[0.5, 0.5]Balanced — START HERE ★Mixed query types; you haven't measured yet
[0.3, 0.7]More vectorMostly semantic / natural-language queries

"Start at 50/50. Tune based on your query patterns." — slide 05

The source contradicts itself on the starting value, and you should know which to follow. Slide 05 and the walkthrough both say start at [0.5, 0.5] 02:05:03, 02:12:09. The committed advanced_rag.py ships [0.4, 0.6] — semantic-leaning. Neither is wrong; the slide is the pedagogical default and the repo is one developer's tuned result for a corpus of eight prose documents where BM25 has little to bite on. Start at 50/50, then move.

Also note what the weights do not do. They are multipliers on w/(rank + 60), a term that varies by ~3% across the top of a list. Moving from [0.5, 0.5] to [0.7, 0.3] is a much larger perturbation than moving a document from rank 1 to rank 3. Weights are a coarse lever; if the right document is not in either branch's candidate set, no weight rescues it. That is a k problem, not a weight problem.

The head-to-head harness

You do not tune weights by reasoning. You tune them by running the same query through all three retrievers and reading which one put the right document at rank 1 02:06:04. Write this once and reuse it for every retrieval change you ever make:

Python
# RECONSTRUCTED — the on-screen harness file is not in either published repo. The shape,
# the five queries and their category labels are transcribed from the walkthrough; the
# retriever APIs are the verified ones above. Bind the three retrievers however your code
# names them (in `advanced_rag.py` the vector branch is called `semantic_retriever`).
def test_query(retriever, query, name, n=3):
    docs = retriever.invoke(query)
    print(f"\n[{name}] {query}")
    for i, doc in enumerate(docs[:n], start=1):
        src = doc.metadata.get("source", "?")
        print(f"  {i}. ({src}) {doc.page_content[:80]}...")


QUERIES = [
    ("SKU-7742X specifications",  "product code"),
    ("E_CONN_REFUSED error",      "error code"),
    ("How do I authenticate?",    "semantic question"),
    ("WCAG compliance",           "acronym"),
    ("router configuration",      "config lookup"),
]

for query, category in QUERIES:
    print(f"\n===== {category.upper()} =====")
    test_query(bm25_retriever,     query, "BM25")
    test_query(vector_retriever,   query, "VECTOR")
    test_query(ensemble_retriever, query, "HYBRID")

The result of running exactly that in the course, on the video's adversarial corpus, with five queries across three retrievers 02:07:04–02:10:05:

#QueryCategoryVectorBM25HybridInstructor's verdict
1SKU-7742X specificationsproduct code#1 correct, #2/#3 related#1 correct, then noise (WCAG, router doc)#1 correct"vector and hybrid kind of won"
2E_CONN_REFUSED errorerror code#1 correct, #2 also good, #3 slight noise#1 correct, WCAG noise below#1 correct"the clear winner… is the vector"
3"How do I authenticate?"semantic question#1 correctfailed — #1 was WCAG 2.1 compliance#1 correct"vector and hybrid won"
4WCAG complianceacronymcorrectcorrect (keyword nailed it)correct"all of them"
5router configurationconfig lookup#1 correct#1 correct + a related networking doc#1 correctall good

"Vector tends to understand meaning, BM25 matches exact terms, and hybrid gets you the best of both worlds." 02:10:05

Now read queries 1 and 2 again, because they carry the least obvious lesson in the table. Those are the product code and error code queries — the exact cases hybrid exists for — and pure vector search handled both. Why? Because the demo corpus is a handful of documents, so SKU-7742X had almost nothing to be confused with. The value of the BM25 branch grows with corpus size, because the failure it prevents is the right document being outranked by plausible neighbours, and plausible neighbours are exactly what scale gives you (the 10 → 10,000 document cliff from Part 1). A hybrid A/B on a toy corpus will under-report the benefit; run it on production-sized data before you conclude anything.

When to add hybrid, and when not to

Read the right-hand column as a latency argument, not a quality one — the SKIP cases are not cases where BM25 would hurt accuracy, they are cases where the ~20–50 ms and the extra moving part buy nothing you need.
Figure 5Read the right-hand column as a latency argument, not a quality one — the SKIP cases are not cases where BM25 would hurt accuracy, they are cases where the ~20–50 ms and the extra moving part buy nothing you need.
Add hybridSkip hybrid
Enterprise data with codes / IDsSimple Q&A chatbot
Technical documentationCreative-writing assistant
Legal documents (statute numbers)Quick prototypes
Mixed query typesLatency-critical paths (adds ~20–50 ms)
Accuracy is critical
"Hybrid RARELY hurts, OFTEN helps""Pure vector is fine here"

"In production with real users? → Add hybrid search. The accuracy boost is worth it." — slide 04, echoed at 02:00:59

The cost of hybrid is time, not correctness 02:00:59. You are running two retrievals instead of one. Budget ~20–50 ms and note that this is additive to your embedding call, your vector search, and (if you add them) your compression and generation calls — the three budgets from Part 1 are always in tension, and this one spends latency to buy quality.

Production considerations for hybrid

Four quadrants, but only the top-left is a correctness issue — the other three are tuning. If you implement exactly one thing from this slide, implement the rebuild.
Figure 6Four quadrants, but only the top-left is a correctness issue — the other three are tuning. If you implement exactly one thing from this slide, implement the rebuild.

1. BM25 has no incremental updates — this is the trap

The instructor calls this out twice, a minute apart, and prefixes it with "very very very important" 02:11:06, 02:12:09. You cannot append to a BM25 index. The BM25Okapi vectorizer computes corpus-wide statistics (document frequencies, average document length, the idf floor) at construction time. Adding a document invalidates all of them.

So the failure looks like this: you call vector_store.add_documents(new_docs), everything succeeds, no exception is raised — and the new documents are invisible to the keyword branch forever. Your hybrid retriever slowly degrades into a vector retriever with extra latency, and the degradation is proportional to how much of your corpus arrived after startup. It is the purest example of invariant 4 in this whole guide.

The fix is structural: make the rebuild impossible to forget by putting both writes behind one method.

Python
# The production skeleton described at [02:10:05]-[02:11:06]. The class name and method
# names are as spoken; the file shown on screen (hybrid_search/prod_hybrid_search.py) is
# NOT present in either published repo, so this is reconstructed from the walkthrough
# plus the verified APIs above. `weighted_rrf` is the fallback function from the toggle.
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_community.retrievers import BM25Retriever


class HybridRetriever:
    def __init__(self, documents, collection_name="hybrid", k=4,
                 weights=(0.5, 0.5), embeddings=None):
        # Swappable: "you can change that to whatever you need"
        self.embeddings = embeddings or OpenAIEmbeddings(model="text-embedding-3-small")
        self.documents = list(documents)
        self.k = k
        self.weights = weights

        # No persist_directory => this Chroma collection is in-memory and dies with the
        # process, re-embedding the whole corpus on every boot. Pass one in production.
        self.vector_store = Chroma.from_documents(
            documents=self.documents,
            embedding=self.embeddings,
            collection_name=collection_name,
        )
        self.vector_retriever = self.vector_store.as_retriever(
            search_kwargs={"k": self.k}
        )
        self._build_bm25()

    def _build_bm25(self):
        """BM25 has no incremental update path — it is always rebuilt from scratch."""
        if not self.documents:          # from_documents unpacks a zip(*...) and
            self.bm25_retriever = None  # raises ValueError on an empty corpus
            return
        self.bm25_retriever = BM25Retriever.from_documents(self.documents, k=self.k)

    def search(self, query):
        if self.bm25_retriever is None:
            return self.vector_retriever.invoke(query)
        return weighted_rrf(
            [self.bm25_retriever.invoke(query), self.vector_retriever.invoke(query)],
            weights=list(self.weights),
        )

    def add_documents(self, docs):
        # Give every Document a stable `id`: langchain-chroma calls collection.upsert(),
        # so ids make a replayed ingest idempotent. Without them it mints a fresh UUID
        # per call and a re-run silently duplicates your corpus (the Part 4 rule).
        self.vector_store.add_documents(docs)   # incremental — cheap
        self.documents.extend(docs)
        self._build_bm25()                      # MANDATORY full rebuild — not cheap

The asymmetry in add_documents is the whole lesson, and it is invariant 3 with the sign flipped: the vector store's write is incremental and cheap, the BM25 write is O(corpus) and recurs on every ingest. At 1,000 documents nobody notices. At 500,000 documents, an ingest that fires once per minute will spend most of its life re-tokenizing the corpus, and self.documents is a full in-memory copy of your text on top of the vector store's copy.

Your escalation path, in order:

  1. Batch the rebuild. Accumulate new documents and rebuild BM25 on a schedule (every N minutes / N documents) rather than per call. Accept a bounded staleness window on the keyword branch and write it down — it's a correctness contract, not an implementation detail.
  2. Double-buffer. Build the new BM25Retriever into a local variable and rebind the attribute only when it is complete, so concurrent queries never see a half-built index.
  3. Move keyword search into the database. Postgres tsvector + ts_rank, or your vector DB's native hybrid mode, gives you a genuinely incremental keyword index and deletes this entire problem. That is the Part 10 upgrade (pgvector / Supabase), and this trap is the main reason to take it.

2. k value: retrieve wide, let RRF narrow

k = 4 or higher is the recommendation, "and let RRF sort all of that" 02:12:09; slide 06 compresses it to "Retrieve more, let RRF sort." The intuition is mechanical: fusion needs candidates to reconcile. At k=1 or k=2 there is almost nothing to fuse and you have paid 20–50 ms for a rounding error. Note that in the demo both branches run at k=3 — below the slide's own advice — which is fine for eight documents and not fine for eighty thousand.

A reasonable production shape: k=10–20 per branch into the fusion, then keep the top 3–5 after fusion (and after reranking, below). The fusion stage is nearly free; it is the generation stage that charges you per token.

3. Monitor which retriever contributes

Slide 06's "TUNE WEIGHTS" panel says monitor which retriever contributes, and this is the one item on the slide with no code anywhere in the course. Implement it yourself — it is about six lines, and without it "tune based on your query patterns" is unactionable:

Python
# Attribution: for each query, log which branch supplied each surviving document.
def attribute(fused, bm25_docs, vector_docs, id_key=None):
    def key(d):
        return d.page_content if id_key is None else d.metadata[id_key]

    bm25_keys = {key(d) for d in bm25_docs}
    vector_keys = {key(d) for d in vector_docs}
    return [
        {
            "rank": i,
            "in_bm25": key(d) in bm25_keys,
            "in_vector": key(d) in vector_keys,
            "source": d.metadata.get("source"),
        }
        for i, d in enumerate(fused, start=1)
    ]

Ship that into your structured logs (Part 7) and you can answer, per week: what fraction of top-3 documents came only from BM25? If it is ~0%, your keyword branch is dead weight — either your weights are wrong, your tokenizer is eating your codes, or you genuinely do not need hybrid. If it is 30%, you have quantified the upgrade and you have the evidence that justifies moving the weights.

4. Latency

~20–50 ms, because you are doing two searches instead of one. The two branches are independent and should run concurrently — EnsembleRetriever.arank_fusion does exactly this with asyncio.gather, while the sync rank_fusion runs them in a plain list comprehension. On the sync path you pay the sum; on the async path you pay the max. Use ainvoke in any latency-sensitive service.

"Low effort, HIGH IMPACT upgrade." — slide 06

That badge is fair. Of everything in this part, hybrid search has the best ratio of lines-of-code to quality recovered — three components, no LLM call, no re-embed — provided you get the rebuild right. Skip the rebuild and the effort stays low while the impact turns negative.

Reranking: what the source promises and what it delivers

Time for plain speech. The course names reranking as the fix for failure mode #3 in its opening table, and the companion README lists it in "Part 3 — Optimize for Quality". But there is no reranker anywhere in either published repo. No cross-encoder, no Cohere Rerank, no CrossEncoderReranker — not one call site. The closest the source gets is a neighbouring lesson recommending that contextual retrieval (Part 14) be combined with BM25 hybrid search "for best results". Treat the absence as a gap in the source, not as a signal that reranking is optional: this part gives you a measurement harness and a candidate set, and reranking is the standard second stage over it.

It matters that you understand the distinction, because RRF is frequently and wrongly called reranking:

Rank fusion (RRF)Reranking (cross-encoder)
InputN ranked lists1 candidate list + the query
Sees the query?No — only positionsYes, jointly with each document
ModelNone; pure arithmeticA trained relevance model
CostMicrosecondsOne model pass per candidate
FixesOne branch missed the documentThe right document was retrieved but ranked #7

A bi-encoder (your embedding model) encodes the query and the document separately and compares the two vectors — fast, indexable, and lossy, because the document's vector was computed without ever seeing your query. A cross-encoder feeds (query, document) through the model together and outputs a relevance score. It cannot be indexed, so it can never be your first-stage retriever; it is exclusively a second stage over a small candidate set.

The pinned environment already ships the machinery, even though the course never touches it:

Python
# .venv/.../langchain_classic/retrievers/document_compressors/__init__.py
__all__ = [
    "CohereRerank",
    "CrossEncoderReranker",
    "DocumentCompressorPipeline",
    "EmbeddingsFilter",
    "FlashrankRerank",
    "LLMChainExtractor",
    "LLMChainFilter",
    "LLMListwiseRerank",
]

And the reranker is four lines of real work — there is no mystery in it:

Python
# .venv/.../langchain_classic/retrievers/document_compressors/cross_encoder_rerank.py
class CrossEncoderReranker(BaseDocumentCompressor):
    """Document compressor that uses CrossEncoder for reranking."""

    model: BaseCrossEncoder
    top_n: int = 3

    def compress_documents(self, documents, query, callbacks=None):
        scores = self.model.score([(query, doc.page_content) for doc in documents])
        docs_with_scores = list(zip(documents, scores, strict=False))
        result = sorted(docs_with_scores, key=operator.itemgetter(1), reverse=True)
        return [doc for doc, _ in result[: self.top_n]]

Note the shape: a reranker in LangChain is a BaseDocumentCompressor, so it plugs into the same ContextualCompressionRetriever wrapper as the extractor below. That is the composition trick that makes the whole family interchangeable — and it means adding a reranker to the hybrid pipeline you just built is a wrapper, not a rewrite:

Python
# Wiring a reranker over the hybrid retriever. NOTE: `HuggingFaceCrossEncoder` needs
# `sentence-transformers`, which the course's pinned environment does NOT install —
# `uv add sentence-transformers` first, and expect a model download on first run.
from langchain_classic.retrievers import ContextualCompressionRetriever
from langchain_classic.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

reranked = ContextualCompressionRetriever(
    base_compressor=CrossEncoderReranker(
        model=HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base"),  # the default
        top_n=3,
    ),
    base_retriever=ensemble_retriever,   # wide first stage: k=10-20 per branch
)

Two things that snippet does not hide. The cross-encoder runs locally on CPU unless you give it a GPU, so its per-candidate cost is latency you own rather than an API bill — which is exactly why the wide-then-narrow k budget matters. And CrossEncoderReranker is declared with extra="forbid", so a mistyped field name raises at construction instead of being silently ignored.

Since the course gives you no numbers for reranking, do not invent any, and be suspicious of anyone who quotes you one without naming the dataset. What the course does give you is a cheaper, LLM-based approximation of the same idea, and that is two sections down.

Parent-document retrieval: search small, return large

This is the chapter's answer to a genuine dilemma you met in Part 3 and could not resolve there, because it is not solvable at chunking time.

Search precisionLLM context completeness
Small chunks only (200 chars)High — focused embedding, sharp matchLow — fragmented; "the answer might reference something in a previous or next paragraph" 02:53:17
Large chunks only (2,000 chars)Low — diluted embedding; "less precise matching and it might miss the best match" 02:54:18High
Parent-documentHighHigh

"Small chunks for search and large context, because small chunks are going to be connected to their parent, which is a large chunk." 02:50:15

The instructor introduces it with the strongest endorsement in the chapter — "if you're serious about RAG quality, this is the upgrade that matters most" 02:50:15. Take that as enthusiasm rather than a measurement: unlike the chunking A/B in Part 3, this pattern is never benchmarked in the source. What is demonstrated is the mechanism, and the mechanism is sound.

The mechanism is two splitters and two stores. Only the children are embedded and indexed; the parents live in a plain key→document store and are fetched by id after the search. The child's job is to be findable; the parent's job is to be readable.

Python
# production-course-main-code/advanced_rag.py — demo_parent_document_retriever() (abridged)
from langchain_classic.retrievers import ParentDocumentRetriever
from langchain_classic.storage import InMemoryStore

# Splitters
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)

# Storage
vectorstore = Chroma(
    collection_name="parent_child_demo",
    embedding_function=OpenAIEmbeddings(model="text-embedding-3-small"),
)
store = InMemoryStore()

# Create retriever
retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

# Add document — this does the splitting AND both writes
retriever.add_documents([long_doc])

query = "What is LangGraph used for?"

# Regular retrieval (would get small chunks)
child_docs = vectorstore.similarity_search(query, k=1)
print("\n--- Child Chunk (what search found) ---")
print(f"Length: {len(child_docs[0].page_content)} chars")

# Parent retrieval (gets full context)
parent_docs = retriever.invoke(query)
print("\n--- Parent Chunk (what's returned) ---")
print(f"Length: {len(parent_docs[0].page_content)} chars")

On the demo document (ai_agents_guide.md, a Markdown guide with ## chapters) and the query "What is LangGraph used for?", the child chunk that search matched was 173 characters and the parent returned to the LLM was 675 characters 02:52:16– 02:53:17. Printing both, side by side, is what makes the mechanism legible — do it once in your own pipeline and you will never confuse the two again.

Three more things the shipped code tells you that the walkthrough does not:

  • The link is a metadata field. ParentDocumentRetriever extends MultiVectorRetriever, whose id_key defaults to "doc_id", and add_documents stamps _doc.metadata[self.id_key] = _id onto every child before indexing. Retrieval is therefore: search children → collect their doc_ids → docstore.mget(ids). That field is also your natural id_key for RRF dedup if you put parent-doc behind a hybrid ensemble.
  • parent_splitter is optional. Omit it and, per the field's own docstring, "the parent documents will be the raw documents passed in" — i.e. the LLM gets the entire source document. That is a fine choice for short documents and a context-window catastrophe for long ones.
  • InMemoryStore is a demo shortcut 02:51:15 — chosen, in the instructor's words, "that way we don't have to deal with a lot". Parents vanish on process exit while a persisted Chroma collection survives, which leaves you with an index full of orphaned children whose doc_ids resolve to nothing. Production needs a durable docstore (a file store, Redis, or a documents table keyed by doc_id).

The cost of the pattern is storage: you keep the parents and the children. A later chapter's technique comparison table (fcc-production-rag-part-6/03_late_chunking.py:351) rates parent-child ★★★ on context quality and ★★☆ on storage, spelling the penalty out as 2× vector store. The instructor's assigned exercise is the right one: "go ahead and find perhaps larger documents and test this out" 02:54:18 — on the tiny demo doc, the benefit is only suggestive, and you should not adopt a 2× storage cost on a suggestion.

Contextual compression: pay at retrieval to save at generation

Parent-document retrieval makes the context bigger. Compression makes it smaller. They are not contradictory — they operate on different diseases. Parent-doc fixes fragments; compression fixes filler.

Contextual compression is extraction, not summarisation 02:55:18. Each retrieved chunk is sent to an LLM with the query, and the model returns only the spans that answer it. You can read the exact prompt in the installed package, and it settles the "does it paraphrase?" question definitively:

Python
# .venv/.../langchain_classic/retrievers/document_compressors/chain_extract_prompt.py
# (the first paragraph is one long line in the source; wrapped here to fit the page)
prompt_template = """Given the following question and context, extract any part of the
context *AS IS* that is relevant to answer the question. If none of the context is
relevant return {no_output_str}.

Remember, *DO NOT* edit the extracted parts of the context.

> Question: {{question}}
> Context:
>>>
{{context}}
>>>
Extracted relevant parts:"""

{no_output_str} is filled with NoOutputParser.no_output_str, the literal string "NO_OUTPUT"; the parser turns that answer into an empty string, and LLMChainExtractor.compress_documents skips any document whose output has length zero. So compression is simultaneously an extractor and a filter: irrelevant chunks do not come back shorter, they do not come back at all.

The wiring is a decorator over any retriever:

Python
# production-course-main-code/advanced_rag.py — demo_contextual_compression() (abridged)
from langchain_classic.retrievers import ContextualCompressionRetriever
from langchain_classic.retrievers.document_compressors import LLMChainExtractor

vectorstore = create_base_vectorstore()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Create compressor
compressor = LLMChainExtractor.from_llm(llm)

# Wrap retriever with compression
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
)

query = "What frameworks exist for building LLM applications?"

# Without compression
base_docs = vectorstore.as_retriever(search_kwargs={"k": 2}).invoke(query)
for doc in base_docs:
    print(f"Length: {len(doc.page_content)} chars")

# With compression
compressed_docs = compression_retriever.invoke(query)
for doc in compressed_docs:
    print(f"Length: {len(doc.page_content)} chars")

Two asymmetries in that snippet are easy to miss and both matter. The baseline runs at k=2 and the compressed path at k=4, so the "before" and "after" lists are not the same documents — it is a demonstration, not a controlled comparison. And because it takes a base_retriever, ContextualCompressionRetriever composes over anything — hybrid ensembles, multi-query, parent-document. That single design decision is why the reranker above is the same interface.

The demo that looked broken, and why that lesson is the valuable one

First run, on the clean TECH_DOCS corpus: 203 → 203 chars and 245 → 243 chars 02:57:22. Essentially nothing happened.

"It may look like it didn't work. It actually did work… the test data that we have is already pretty clean and relevant." 02:57:22–02:58:24

If every retrieved chunk is already 100% on-topic, there is nothing to compress. This is the most useful negative result in the chapter, because it tells you exactly when compression earns its keep and when it is pure cost. Chunks that are short and topically pure — the natural output of good recursive chunking on well-structured documentation — have no filler to remove.

Watch the call volume while you run it. The demo makes two retriever calls — the k=2 baseline and the k=4 compressed path — so two query embeddings, and then one completion per chunk the compressed retriever returned. The instructor counts them off the terminal at 02:56:20, but the captions garble the tally, so take the count from the code rather than the audio: compress_documents loops over the candidate list exactly once, one invoke per document. Compression is the first technique in this guide where the retriever stops being pure vector math and starts costing inference.

To make it visible he authored INFO_BURIED: two long documents — acme_company_overview.pdf and technical_docs_v2.4.pdf — where a couple of relevant paragraphs are buried in company history (founded 2018, Palo Alto garage, offices in Austin and Seattle, $2M → $45M revenue, Series B at a $500M valuation, Aetna health insurance) and infrastructure detail (AWS EKS, Istio, DataDog, RTO 4 hours, RPO 1 hour). Two paragraphs — one about LangChain, one about LangGraph — actually answer the query. Then he pointed the vector-store factory at INFO_BURIED instead of TECH_DOCS and changed nothing else.

Second run: 1,703 chars → 214 chars, and a second document of "over 1,500" chars collapsing similarly. He calls it "about 87 to 82% reduction" 03:00:26, quoting the range backwards; 1,703→214 is ≈87.4%, and the second is ≈82%. He then summarises it as "85% fewer tokens in our final prompt" 03:01:29. All three figures are the same measurement on one hand-built demo document, not a benchmark — treat them as directional.

What disappeared is as instructive as the percentage: before compression the context carried the founding story, the office locations and the entire AWS architecture chapter; after it, only the LangChain and LangGraph paragraphs survived 03:00:26. The extractor behaved exactly as its prompt instructs — it read each chunk, asked "what here answers what frameworks exist for building LLM applications?", and returned those spans verbatim.

The three claimed benefits 03:01:29–03:02:31 — and the honest accounting on each:

BenefitMechanismThe catch
Lower cost~85% fewer context tokens in the generation promptYou spend LLM tokens at retrieval time to save them at generation time
Better answersLess noise; "no risk of the LLM getting distracted by company history or infrastructure details"Only true when there was noise
FasterSmaller context → faster inference, "especially noticeable with larger documents"You added one LLM round-trip per retrieved chunk before generation

That last catch is the whole trade-off, stated plainly at 03:02:31: "extra calls to the completion… which means LLM cost at retrieval time." With k=4 you make four extra LLM calls per user query. compress_documents runs them in a serial for loop on the sync path; acompress_documents uses llm_chain.abatch, so the async path is dramatically faster for the same spend. Use ainvoke.

The three conditions where compression is worth it 03:02:31:

  1. Documents are large with mixed content.
  2. You need precise, focused retrieval.
  3. Token costs matter at scale.

Invert that and you get the skip rule: small clean chunks, latency-sensitive path, low volume → do not compress.

One failure mode to guard: if the extractor returns NO_OUTPUT for every candidate, your retriever returns zero documents and your chain formats an empty context string. The LLM then answers from parametric memory unless your prompt has the Part 1 fallback clause. Always test the empty-context path.

Multi-query: fix the query, not the index

Hybrid search fixes queries whose tokens are wrong for a vector index. Multi-query fixes queries whose phrasing is wrong: one user question becomes N rephrasings, each is retrieved for, and the union is de-duplicated 03:06:38. The de-duplication is not bookkeeping — it is where the value shows up, because a document that surfaces for three different phrasings of the question is a document three independent probes agree on.

This is also the answer to the question everyone asks when they see the constructor: why does a *retriever* need an llm= argument? 03:07:39 Because generating the variations is an LLM call. Multi-query is the first retriever in this part that costs a completion before it touches the index.

The generation prompt is not a mystery either — it is a constant in the package:

Python
# .venv/.../langchain_classic/retrievers/multi_query.py
DEFAULT_QUERY_PROMPT = PromptTemplate(
    input_variables=["question"],
    template="""You are an AI language model assistant. Your task is
    to generate 3 different versions of the given user
    question to retrieve relevant documents from a vector  database.
    By generating multiple perspectives on the user question,
    your goal is to help the user overcome some of the limitations
    of distance-based similarity search. Provide these alternative
    questions separated by newlines. Original question: {question}""",
)

Three variations, hard-coded in the default template — that is where the "3" in the walkthrough comes from, not from a parameter. Pass your own prompt= to change it. The output is parsed by LineListOutputParser, which splits on newlines and drops blanks, so a model that answers with a numbered list will hand you variations prefixed 1. , 2. — harmless for retrieval, ugly in logs.

Python
# production-course-main-code/advanced_rag.py — demo_multi_query_retriever() (abridged)
from langchain_classic.retrievers.multi_query import MultiQueryRetriever

# Enable logging to see multi-query generation  (module top, lines 25-27)
logging.basicConfig(level=logging.INFO, format="%(name)s - %(message)s")
logging.getLogger("langchain.retrievers.multi_query").setLevel(logging.INFO)

vectorstore = create_base_vectorstore()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.3)

# Create multi-query retriever
retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(search_kwargs={"k": 2}), llm=llm
)

query = "What tools can I use to build AI applications?"
print("(Check INFO logs above for generated queries)\n")

docs = retriever.invoke(query)
print(f"Retrieved {len(docs)} unique documents:")

Points worth internalising:

  • temperature=0.3, not 0. You want variation in the rephrasings. This is one of the few places in the whole guide where a non-zero temperature is the correct default.
  • You must turn on INFO logging to see the rewrites 03:04:38. By default they are invisible, which means you are paying for query expansion with no visibility into whether the expansions are sane. That is unacceptable in production.
  • The cost is N embedding calls plus one completion, per user query. With three variations the run made three calls to the embeddings endpoint and returned 3 unique documents 03:08:40. The k=2 per variation means up to 6 hits collapsing to 3 after dedup.
  • Deduplication is exact-object equality. _unique_documents is [doc for i, doc in enumerate(documents) if doc not in documents[:i]] — a Document comparison, so identical text with differing metadata survives as two entries. Combine with parent-doc or a metadata-stamping loader and you can get near-duplicates in context.
  • include_original defaults to False. The user's literal question is not among the retrieved-for queries unless you ask for it. If your users type precise technical queries, set include_original=True — otherwise you have replaced their exact wording with an LLM's paraphrase, which is precisely the thing RunnablePassthrough exists to prevent in the generation half of the chain.

Multi-query is a recall tool: it exists to get the right document into the candidate set. Pair it with a precision tool (compression or a reranker) or you have simply made your context noisier.

Composing the stack

Every technique in this part is a BaseRetriever in and a BaseRetriever out, which means they nest. The repo's most advanced demo stacks two of them:

Python
# production-course-main-code/advanced_rag.py — demo_advanced_rag_chain() (abridged)
vectorstore = create_base_vectorstore()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Multi-query for better recall
multi_retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}), llm=llm
)

# Compression to focus on relevant info
compressor = LLMChainExtractor.from_llm(llm)
advanced_retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=multi_retriever
)

prompt = ChatPromptTemplate.from_template(
    """
Answer the question based on the following context. Be specific and cite which technologies you're referring to.

Context:
{context}

Question: {question}

Answer:"""
)

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": advanced_retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Recall first, precision last. Multi-query widens (3 variations × k=3 → up to 9 candidates), compression narrows. Reversing the order would compress documents you are about to discard — pure waste.

Do the arithmetic before you ship this, because the composition is where the money goes. One user question through that chain costs: 1 completion (query generation) + 3 embedding calls + 3 vector searches + up to 9 completions (one per surviving candidate, serial) + 1 completion (the answer). That is up to 11 LLM round-trips for one question. It is a legitimate architecture for a low-volume, high-value internal tool and an obviously wrong one for a public chatbot. Part 9 — which owns both the token budget that rejects a request before it spends any of that, and the caches that stop you paying twice — exists for exactly this chain.

Here is the ordering rule generalised:

StageTechniqueDirectionCost
1Query expansion (multi-query)Widen1 LLM call + N embeddings
2Hybrid retrieval (vector + BM25 → RRF)Widen~20–50 ms, 1 embedding
3Metadata filterNarrowFree (index-side)
4Rerank (cross-encoder)Narrow1 model pass per candidate
5Compression / extractionNarrow1 LLM call per surviving chunk
6Parent expansion (docstore lookup)Widen (deliberately)1 key-value get per hit

Parent expansion goes last on purpose: you want to select on focused children and then hand the LLM whole parents. Selecting on parents defeats the entire pattern.

Citations: making the answer auditable

Retrieval quality that the user cannot verify is a claim, not a result. The trust chain from Part 1 — sources → citations → verifiability → trust — reframes accuracy as auditability: the system's job is to make checking cheap, not to be believed.

The remarkable thing is how little code it takes. Compared with the basic RAG chain, citations are a prompt change and a formatter change, nothing else 03:09:42–03:11:49.

One small refactor comes with it, performed live and preserved in the repo: the LLM is hoisted out of the demo functions to module scope — llm = init_chat_model(model="gpt-4o-mini", temperature=0.2) at line 58 — so every chain in the file shares one client 03:09:42. Minor, but it is the shape you want: one model declaration per module, the same discipline invariant 2 demands of the embedding model.

Python
# production-course-main-code/rag_pipeline.py — demo_rag_with_sources()
def demo_rag_with_sources():

    vectorstore = create_kb()
    retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

    prompt = ChatPromptTemplate.from_template(
        """
Answer the question based on the context below. Include which sources you used.

Context:
{context}

Question: {question}

Answer (include sources):"""
    )

    def format_docs_with_sources(docs):
        formatted = []
        for i, doc in enumerate(docs):
            source = doc.metadata.get("source", "unknown")
            formatted.append(f"[{i+1}] {source}:\n{doc.page_content}")
        return "\n\n".join(formatted)

    rag_chain = (
        {
            "context": retriever | format_docs_with_sources,
            "question": RunnablePassthrough(),
        }
        | prompt
        | llm
        | StrOutputParser()
    )

    answer = rag_chain.invoke("What are the core components of LangChain?")

Three details carry the whole pattern:

  1. Sources travel *inside* the context string. format_docs_with_sources prefixes each chunk with [n] <source>: before the text. The model never receives metadata out of band — it reads the citation label as part of the document, which is why it can reproduce it. doc.metadata.get("source", "unknown") is the defensive form; a KeyError here would kill the request over a missing label.
  2. The output slot does the asking. Answer (include sources): is doing as much work as the instruction sentence above it. Models complete the shape you give them.
  3. The chain skeleton is unchanged. {"context": retriever | formatter, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() — identical to Part 5's basic chain, identical to the advanced chain above.

"The mechanics are always the same. What changes is perhaps a few things." 03:11:49

That is the structural takeaway of this entire part. Hybrid, multi-query, compression, parent-doc and citations are five different edits to one of the six slots in that expression. Nothing else moves.

In the run, the answer came back with the five core LangChain components — models, prompts, chains, agents and memory — cited to langchain_knowledge_base.md, the source stamped onto the document back in create_kb():

Python
# production-course-main-code/rag_pipeline.py — create_kb()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
doc = Document(
    page_content=KNOWLEDGE_BASE, metadata={"source": "langchain_knowledge_base.md"}
)
chunks = splitter.split_documents([doc])

split_documents propagates that metadata onto every chunk. Had the code used split_text, the source would have been destroyed at chunking time and no amount of prompt engineering could have recovered it. Citability is decided at index time — the same rule as filterability in Part 4.

"If you're building a Q&A bot, having this citation here, it's really important." 03:11:49

The reason he gives is the product one: a citation lets the user click through and see where the information actually came from.

The limits of prompt-based citations

Be clear-eyed about what you have built: the citation is generated text. The model is copying a label it read in the prompt, and models miscopy, merge and invent labels. Nothing in this pattern enforces that [2] corresponds to the document that actually supported the sentence.

Two hardening moves, both using code you already have:

  • Return the retrieved documents alongside the answer, so your UI links to the real chunk objects rather than parsing the model's prose. Build the chain with RunnableParallel — already imported in rag_pipeline.py — and keep the docs branch.
  • Ask for structure instead of prose. The same repo file already does this, and the sources_used field is machine-checkable — you can assert every returned source appears in the retrieved set and flag the request when it doesn't:
Python
# production-course-main-code/rag_pipeline.py — demo_structured_rag()
# (RAGResponse is declared inside the demo function in the repo)
class RAGResponse(BaseModel):
    """Structured RAG response."""

    answer: str = Field(description="The answer to the question")
    confidence: str = Field(description="high, medium, or low")
    sources_used: List[str] = Field(description="List of sources referenced")
    follow_up: str = Field(description="Suggested follow-up question")

structured_llm = llm.with_structured_output(RAGResponse)

Note that confidence is the model's self-report typed as a free string, not a calibrated probability and not even a constrained enum. It is a useful sorting signal for human review and worthless as a correctness guarantee — the Part 1 rule stands: coherence is not correctness, and neither is confidence.

Two citation traps carried forward from Part 2 that bite hardest here:

  • page is 0-indexed, page_label is a 1-indexed string. Cite page and every citation in your product is off by one.
  • Documents loaded from tempfile paths carry garbage source metadata. create_kb() uses tempfile.mkdtemp() for the Chroma persist directory (fine, it's storage), but if you ever load documents from temporary paths, the source you cite is a path that no longer exists. Load from stable paths whenever you intend to cite.

Provenance: what is verified, what is not

This part draws on a chapter whose captions are auto-generated and a repo that does not match the video in several places. Everything above is checked against the shipped code where shipped code exists. Here is what isn't.

Flagged discrepancies and unverified details
  • hybrid_search/prod_hybrid_search.py — the file on screen for the entire hybrid walkthrough 02:02:59 does not exist in either published repo (production-course-main-code, fcc-production-rag-part-6). The closest real code, and the source of every hybrid snippet above, is production-course-main-code/advanced_rag.py::demo_ensemble_hybrid_search(). The SKU / error-code corpus and the five-query results table come from that missing file, not from the repo's TECH_DOCS.
  • The HybridRetriever class 02:10:05 is spoken, never shown. The method names search() and add_documents() are as narrated; the skeleton above is a reconstruction built on verified APIs, not a transcription.
  • chunking/semantic_chunking.py and chunking/production_ready.py — also on screen, also absent from both repos. They belong to a third repo ("Part 3 — Optimize for Quality"). Relevant to Part 3, not here, but it is the same missing-artifact pattern.
  • XR9000 — spoken once as a second product code 01:53:47 with no slide corroboration; slide 02 shows different distractors (Toaster X-500, Tool-88, Per-98, Shoe-99). Illustrative only.
  • [0.5, 0.5] vs [0.4, 0.6] — slide and walkthrough say the former, committed code ships the latter. Reconciled above: start at 50/50.
  • "Hybrid won on every query type" (slide 06) vs the walkthrough, where vector is repeatedly called the clear winner and hybrid ties. Reconciled above: hybrid was never worse.
  • "87 to 82% reduction" 03:00:26 is quoted high-to-low; the underlying measurements are 1,703→214 (≈87%) and ~1,500→~270 (≈82%, second value never stated aloud). The "85% fewer tokens" at 03:01:29 is the same measurement rounded again.
  • The per-query LLM call count during compression 02:56:20 is unreconstructable from the captions ("1 2 3 4 five calls to embeddings and also the completion"). The claim above — one completion per retrieved chunk — is read from LLMChainExtractor.compress_documents, not from the audio.
  • The five-query evaluation harness is a reconstruction. The queries and their category labels are transcribed from 02:06:04–02:10:05; the test_query helper's exact name and signature are not, and the source file is the missing prod_hybrid_search.py.
  • advanced_rag.py's docstring advertises "self-query" and the instructor lists it in the chapter agenda 03:03:34, but there is no SelfQueryRetriever anywhere in either repo and it is never demoed. This part does not cover it, and neither does the course.
  • The reranker wiring snippet above is composed from verified APIs (CrossEncoderReranker in langchain_classic, HuggingFaceCrossEncoder in langchain_community), not copied from the course — and sentence-transformers, which HuggingFaceCrossEncoder imports at construction, is absent from the pinned environment.
  • The RRF constant c = 60 is never spoken. It is read directly from langchain_classic/retrievers/ensemble.py:70, i.e. the algorithm the instructor re-implemented by hand.
  • No reranker exists in the source material. The CrossEncoderReranker snippet above is read from the installed langchain_classic package, not from the course, and the course quotes no reranking benchmark. Anthropic's contextual-retrieval figure that the course does quote — 67% fewer retrieval failures — is attributed in the repo to contextual retrieval, and belongs to Part 14, not here.
  • The multi-query logger name in the repo (langchain.retrievers.multi_query) does not match the module's actual logger (langchain_classic.retrievers.multi_query), verified by reading multi_query.py and confirming that the pinned langchain package has no retrievers submodule.
  • The repo's citation query is "What are the core components of LangChain?"; the video says "common components". Same question, different word.

Where this part hands off

  • Token budgeting — the second half of the same chapter marker 02:13:12–02:21:10 pivots to TokenBudget / BudgetedLLM. Compression, multi-query and hybrid are exactly the features that make a pre-flight token gate necessary. Part 9.
  • Database-side hybrid search. Everything here runs in-process. pgvector/Supabase can do the keyword half inside the database with an incremental index, deleting the BM25 rebuild problem entirely. Part 10.
  • Contextual retrieval. Prepending LLM-written context to each chunk at index time is a different technique from contextual compression — do not conflate the names. The course's own key-takeaway line for it is "combine with BM25 hybrid search for best results" (fcc-production-rag-part-6/02_contextual_retrieval.py:473), i.e. these two stack. Part 14.
  • Parent-child cost accounting — the ★★★-context / ★★☆-storage, 2×-vector-store rating — and the late-chunking alternative that tries to buy the same context quality without the second copy. Part 14.
  • Agentic retrieval — evaluate the retrieved set with an LLM grader and re-query when it is bad, which is what you reach for when reranking and compression still aren't enough. Part 15.
  • Multi-hop questions, where the bridging entity never appears in the query. Neither branch of a hybrid retriever can help. Part 16.

The technique matrix

TechniqueFixesPhaseAdded cost per queryAdd it when
Hybrid (BM25 + RRF)Codes, IDs, acronyms, exact namesRetrieve~20–50 ms, no LLMYou're in production with real users and your corpus has identifiers
Weight tuningWrong branch dominatingFuseFreeAfter you have query-category data
Reranking (cross-encoder)Right doc retrieved, ranked #7Rerank1 model pass per candidateRecall is fine but precision@3 is not
Parent-documentFragmented context; diluted embeddingsRetrieve1 docstore get per hit; 2× storageChunks are precise but answers reference neighbours
Contextual compressionFiller crowding out signalCompress1 LLM call per retrieved chunkBig, mixed documents and token costs matter
EmbeddingsFilterSame, cheaplyCompress1 query + N candidate embeddings, no LLMYou want compression's filtering without its bill
Multi-queryPhrasing mismatchQuery1 LLM call + N embeddingsUsers ask the same thing many ways
CitationsUnverifiable answersAssemble~0Always

Decisions to make

  1. Do you need hybrid at all? The source's rule is qualitative — codes, IDs, statute numbers, mixed query types, accuracy critical. Make it quantitative: sample 200 real queries and count how many contain a code, an ID, an acronym, a proper name or a quoted exact phrase. That fraction is your expected hit rate for the BM25 branch, and it is the number you will later compare against the attribution logs. (A ~10% trigger is this guide's rule of thumb, not the course's; what matters is that you measured the fraction at all.)
  2. In-process BM25 or database-side keyword search? rank_bm25 is in-memory, O(corpus) per query, and rebuild-only. It is right up to roughly the tens of thousands of documents and wrong beyond that. Decide now, because migrating later means re-architecting ingest.
  3. What is your BM25 tokenizer? The default is str.split(). If your identifiers carry punctuation — and they do — write a preprocess_func, pass it to from_documents (not to the instance afterwards), and unit-test it against ten real codes from your corpus.
  4. What are your two k values? Per-branch k into fusion (start 10–20; the course's floor is 4 or higher) and post-fusion k into the prompt (3–5). They are different numbers and conflating them is the most common tuning error.
  5. Where do the weights start? [0.5, 0.5], written explicitly — omitting weights silently gives you uniform weights, which looks the same until someone adds a third retriever. Keep the retriever list and weight list on adjacent lines with the percentages in a comment, and move only on attribution data.
  6. How will you rebuild BM25? Per-write, batched, or double-buffered — and what is the documented staleness window for the keyword branch?
  7. Rerank or compress? They solve different halves of the noise problem. Reranking reorders (cheap per candidate, no text change). Compression shrinks (expensive, rewrites content, can return zero documents). Most systems want a reranker first and only add compression if the surviving chunks are still bloated.
  8. Is your compression demo honest? If your test corpus is clean, compression will look broken. Build an INFO_BURIED-style document with real filler before you decide it does nothing.
  9. Where does the parent live? InMemoryStore is a demo. Name the durable docstore and the doc_id key before you index anything, because orphaned children are unrecoverable without a re-index.
  10. Does the user's literal question survive? Multi-query with include_original=False means it doesn't. Decide deliberately.
  11. How do you verify citations? Prose citations are model-generated and unverified. Either return the document objects alongside the answer or use structured output with a sources_used field you can assert against the retrieved set.

Checklist before you call retrieval "done"

Each line is written so it either passes or fails — no line here is satisfied by an opinion.

  • A test_query(retriever, query, name) harness exists, is checked in, and runs the same query through every retriever variant side by side. Running it is one command.
  • Test queries are annotated by category (product code, error code, semantic, acronym, config lookup), and the fixture corpus is adversarial: it contains real codes, IDs and acronyms from your domain, not eight paragraphs of clean prose.
  • Retrieval is measured separately from generation — you can answer "did the right chunk come back?" without reading an answer.
  • There is a test that calls add_documents() with a document containing a unique token, then asserts the BM25 branch alone returns it. This is the one test that catches the silent hybrid-to-vector decay.
  • The BM25 preprocess_func is passed to from_documents(...) at construction (never assigned on the instance), and has a unit test over ten real identifiers from your corpus, including one with trailing punctuation and one in the wrong case.
  • The ensemble's weights are written explicitly on the line below retrievers, in the same order, with the percentages in a comment. Nobody is relying on the silent 1/n default.
  • id_key is set to a real document identifier if two chunks in your corpus can share text; otherwise you have confirmed the page_content dedup key is safe, and that every document carries the id_key field (a missing one raises KeyError mid-fusion).
  • The two k values are distinct and written down: per-branch k into fusion, post-fusion k into the prompt.
  • Retriever attribution (BM25-only / vector-only / both) is in your structured logs (Part 7), and someone has read last week's numbers and can state the BM25-only percentage from memory.
  • Async paths (ainvoke) are used wherever two retrievals or N compressions run; the sync path pays the sum instead of the max.
  • Multi-query's generated variations are logged as structured fields — not via a logger name copied from a tutorial, which in the pinned environment configures a logger that does not exist.
  • The empty-context path is tested: compression returning zero documents must produce a refusal, not a hallucinated answer.
  • Parent documents live in a durable docstore, not InMemoryStore, and there is a check that every indexed child's doc_id resolves to a parent.
  • source metadata survives from loader to chunk to prompt (split_documents, not split_text), and citations use page_label, not page.
  • Cited sources are validated against the retrieved set — either by returning the Document objects alongside the answer, or by asserting sources_used ⊆ retrieved sources on structured output.
  • Every quality upgrade has a measured before/after on your corpus. Every figure in this part — ~20–50 ms, 87%, 85%, 173 → 675 chars — is one demo on one document, in one run, by one person. None of them is your number until you have measured it.