Embeddings, Vector Stores and Reading Similarity Scores
Turn chunks into vectors, store them in Chroma, and learn to read similarity scores correctly.
Part 3 left you with chunks. Chunks are text, and text cannot be compared arithmetically. This part is the bridge: it turns those chunks into numbers, puts the numbers somewhere you can search them, and then — the bit almost everyone gets wrong the first time — teaches you to read the numbers that come back out.
In the mental model from Part 1, you are working on exactly four boxes:
Original ASCII
PHASE A — INDEX Load → Split → [Contextualize] → [EMBED] → [STORE + INDEX] PHASE B — QUERY Gate → Cache → [EMBED QUERY] → [RETRIEVE] → Fuse/Filter → …
Two of those boxes are in the offline phase and are paid once. Two are in the online phase and are paid on every single request. The same embedding model has to sit in both — that is invariant 2, and it is the load-bearing rule of this part. Everything else here is mechanics.
Source: chapters 6, 7 and 8 of the course, 00:48:12 – 01:24:32 — a slide-heavy theory chapter, the first hands-on chapter of the whole course, and a very short chapter that is really two lessons glued together.
An embedding model is not a chat model
The instructor opens chapter 6 by calling this out as "this very crucial distinction that confuses a lot of people" 00:48:13, and he is right to. Both live behind the same OpenAI client object, both are billed per token, both are called "models". They are not interchangeable in either direction.
| Chat model | Embedding model | |
|---|---|---|
| Input | a conversation (messages=[{role, content}]) | one plain string (input="…") |
| Output | text you can read | a list of floats |
| SDK surface | client.chat.completions.create(...) | client.embeddings.create(...) |
| Example model | gpt-4o-mini | text-embedding-3-small |
| Read the result from | response.choices[0].message.content | response.data[0].embedding |
| Swappable later? | Yes — freely | No — see "The Rule" below |
00:49:14 — "So same company, different model."
The chapter's teaching move is deliberately minimal: he writes the chat demo, runs it, comments it out, uncomments the embeddings demo directly beneath, and runs that. Three lines change. The output type changes completely.
# Reconstructed from the ch.6 narration [00:49:14]–[00:51:16].
# NOT a checked-in repo file — the instructor types this into a scratch buffer.
from openai import OpenAI
client = OpenAI()
# --- demo 1: CHAT model ---
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.choices[0].message.content) # -> "The capital of France is Paris."
# --- demo 2: EMBEDDING model (the chat block above gets commented out) ---
response = client.embeddings.create(
input="Your text string goes here",
model="text-embedding-3-small",
)
print(response) # full object: vector + metadata
print(response.model) # "text-embedding-3-small"
print(response.object) # "list"
print(response.usage) # prompt_tokens / total_tokens
print(len(response.data[0].embedding)) # 1536
Three things in that second block are worth more than they look.
len(response.data[0].embedding) → 1536. That is the entire definition of dimension, delivered as a hands-on measurement rather than a slide bullet 00:51:16. The dimension of text-embedding-3-small is 1536 because the list it returns is 1536 floats long. Nothing mystical.
response.usage carries prompt_tokens and total_tokens. Embedding calls are billed on tokens exactly like chat calls 00:50:14. People internalise "chat costs money" and somehow assume embeddings are free. They are not, and — see below — you pay for them twice: once per chunk at index time and once per question forever.
Why numbers at all? Because numbers can be compared mathematically and text cannot 00:51:16. You can compute a distance between two vectors. You cannot compute a distance between two English sentences. Everything downstream — similarity search, thresholds, reranking, fusion — exists only because the representation became arithmetic.
00:51:16 — "Because numbers can be compared mathematically easily."
Choosing the dimension
More dimensions means more room to store features about whatever you embedded — more meanings, more semantics, more distinguishable nuance 00:52:18. That is the only thing that gets better as the number goes up. Three things get worse.
00:54:20 — "More dimensions equals more semantic nuance captured. But more storage and slower search."
The reference table. Note that the dimensions come from the slides 00:52:18–00:53:19 and the prices come from a comment block at the top of embeddings.py in the repo — the instructor never says a price out loud, so treat the money column as repo-sourced and verify it against current provider pricing before you quote it to anyone.
| Model | Dimensions | Cost / 1M tokens | Best for |
|---|---|---|---|
text-embedding-3-small | 1536 | $0.02 | General use — the course default |
text-embedding-3-large | 3072 | $0.13 | High accuracy (6.5× the price) |
text-embedding-ada-002 | 1536 | $0.10 | Legacy — no reason to pick it today |
| Gemini embedding model | 768 | free tier | "kind of small but the good news is that it's free" |
| BGE-small | 384 | local / free | Very small; speed and footprint dominate |
all-MiniLM-L6-v2 (Sentence-Transformers) | 384 | local / free | The repo's stand-in for the 384-dim class |
Two honest caveats on that table. First, the instructor names BGE-small on the slide at 00:53:19 but the repo ships sentence-transformers/all-MiniLM-L6-v2 instead — both are 384-dim, both are common small open models, but they are not the same checkpoint; don't let the guide's own table imply they are. Second, he says "Gemini embeddings, 768 dimensions" without ever naming a version string. Google has shipped both text-embedding-004 at 768 and gemini-embedding-001 with configurable dimensions, so resist the urge to fill in a model id that the source never gave.
The decision procedure, in the order you should actually run it:
- Default to the 768–1536 band. Stop here unless something below applies.
- Maximum semantic fidelity, and you can pay the storage and per-query latency forever?
→ 3072 (text-embedding-3-large), at 6.5× the token price.
- Cost is the binding constraint, or you want a free tier? → 768 (Gemini).
- Footprint and speed dominate and the semantics are simple? → 384 (BGE-small,
all-MiniLM-L6-v2, or another local model).
- In all cases: decide from your workload, not from a leaderboard. The refrain through the
whole chapter is "depends on use cases."
Here is the real repo file. It is short, it is mostly a comment, and the comment is the most quotable thing in it — but it also contains three separate traps, so read the annotations before you copy it.
# /home/oriol/lab/rag/code/production-course-main-code/embeddings.py (lines 1-17, verbatim)
# Model Dimensions Cost per 1M tokens Best For
# text-embedding-3-small 1536 $0.02 General use
# text-embedding-3-large 3072 $0.13 High accuracy
# text-embedding-ada-002 1536 $0.10 Legacy
from langchain_openai.embeddings import OpenAIEmbeddings
from langchain_community.embeddings import HuggingFaceEmbeddings
from dotenv import load_dotenv
load_dotenv()
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2") # 384 dimensions
# Ollama
from langchain_ollama import OllamaEmbeddings
embeddings = OllamaEmbeddings(model="llama2-7b-embedding-q4_0")
The rest of the file — commented out in the repo — is the two-method LangChain API you will use for the remainder of this guide:
# /home/oriol/lab/rag/code/production-course-main-code/embeddings.py, lines 33-51,
# with one leading "#" stripped from each line. Otherwise unchanged.
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# single text
text = "This is a sample text to be embedded."
embedding = embeddings.embed_query(text)
# print(f"Embedding for single text: {embedding}")
print(len(embedding)) # Should print 1536 for text-embedding-3-small
# multiple texts
embeds = embeddings.embed_documents(
["This is the first document.", "This is the second document."]
)
print(f"Embeddings for multiple texts: {embeds}")
print(f"Number of embeddings returned: {len(embeds)}") # Should print 2
print(
f"Length of each embedding: {len(embeds[0])}"
) # Should print 1536 for text-embedding-3-small
embed_query is for the single question on the read path. embed_documents takes a list and returns one vector per element in a single round-trip — that is the method you want at index time, and the distinction matters later when you cache, because CacheBackedEmbeddings caches embed_documents and leaves embed_query uncached unless you ask (Part 9). Note also that this block binds the name embeddings a third time: if you simply uncomment it in place, it lands below the two live bindings at the top of the file and quietly wins. Delete the ones you are not using.
What the numbers buy you: ranking by cosine similarity
Before handing the vectors to a database, it is worth doing the search by hand once. Thirty lines of numpy is the whole idea of semantic search, with no store, no index and no abstraction in the way.
# /home/oriol/lab/rag/code/production-course-main-code/embeddings_deep.py (lines 35-65)
def similarity_search():
docs = [
"Python is a programming language",
"JavaScript is used for web development",
"Machine learning enables AI applications",
"Deep learning uses neural networks",
"Cats are popular pets",
]
query = "What programming languages exist?"
# embed documents and query
doc_vector = embeddings_model.embed_documents(docs)
query_vector = embeddings_model.embed_query(query)
# compute cosine similarities
def cosine_similarity(vec1, vec2):
return np.dot(vec1, vec2) / (np.linalg.norm(vec1) * np.linalg.norm(vec2))
similarities = [cosine_similarity(query_vector, doc_vec) for doc_vec in doc_vector]
# rank documents by similarity
ranked_docs = sorted(zip(docs, similarities), key=lambda x: x[1], reverse=True)
print(f"Query: {query}\n")
print("Ranked by similarity:")
for doc, score in ranked_docs:
print(f" {score:.4f}: {doc}")
Read what a vector database actually does, in four lines: embed the corpus, embed the query, score every pair, sort descending. A production vector store adds an index so you do not have to score every pair, plus persistence, filtering and concurrency — but the semantics above are the semantics it implements.
Note the polarity here: this function sorts reverse=True, because cosine similarity is higher-is-better. Hold that thought; Chroma is about to hand you the opposite convention.
One more thing in that file is worth stealing even though it does not run. basic_embeddings() prints three facts about a fresh vector — its length, its first five values, and np.linalg.norm(...). That is the minimum "what did I actually get back" probe, and the third print is the interesting one: if the norm comes back ≈1.0000 your provider is handing you unit-length vectors, in which case cosine similarity and a plain dot product are the same number and the division in cosine_similarity is doing nothing. Do not assume it — print it. Providers differ, and a provider that does not normalise will quietly make your dot-product shortcut wrong.
The two phases, restated concretely
This is the spine of chapter 6 00:54:20–00:56:23, and it is the same two-phase model from Part 1 with the boxes filled in.
Phase 1 — Indexing. Happens once, per document set.
- Load the documents.
- Split them — "let's say 500 to 1,000 tokens each" 00:54:20.
- Embed every chunk through your chosen embedding model.
- Store the result.
What actually lands in the store is not just the vectors: "we have actual vectors, embeddings, original text and all of that" 00:55:23. The vector, the original chunk text and its metadata travel together. That matters because step 3 of the query phase needs the text, not the numbers. At the end of indexing the store holds thousands of vectors, each representing one chunk.
(The instructor uses "vector store" and "vector database" interchangeably and says so explicitly. This guide does the same.)
Phase 2 — Querying. Happens per question, every question.
- Embed the query — "the same embedding model — this is important" 00:55:23 — into
a query vector.
- Search the store for similar vectors.
- Retrieve — "the process of getting the original text chunks for those vectors."
- Augment — combine system prompt + retrieved chunks + user query 00:55:23.
- Generate — send the assembled prompt to a chat model and return the answer.
Step 4 is the "A" in RAG, and it is more specific than people usually state it. Augmentation is exactly three ingredients merged into a prompt 00:55:23. There is no retraining, no fine-tuning, no weight update — "we are augmenting the large language model's knowledge with our retrieved documents" 00:56:23, at prompt time, for one request, and then it is gone.
And the asymmetry that falls out of putting the two phases side by side:
Index-time embedding is a one-off batch cost you amortize. Query-time embedding is paid on every single question, forever. Dimension choice therefore bites hardest at query time — a 3072-dim model is not "twice the storage", it is twice the storage and a slower vector on the critical path of every request.
The fix for the recurring half is caching, which the course puts in embeddings_deep.py::embedding_caching() — CacheBackedEmbeddings.from_bytes_store over a LocalFileStore — and this guide covers properly in Part 9. Three things to know now so you do not build the wrong habit:
- In LangChain 1.x both of those symbols live under
langchain_classic, notlangchain:
from langchain_classic.embeddings.cache import CacheBackedEmbeddings and from langchain_classic.storage import LocalFileStore. Every older tutorial you will find imports them from the top-level langchain package and will ImportError on you.
CacheBackedEmbeddingspopulatesdocument_embedding_cache— i.e. it caches
embed_documents and leaves embed_query uncached unless you also pass query_embedding_cache=. The repo passes only the first, so the index path is cached and the query path, the one you actually pay for on every request, is not.
- The
namespace=argument should be your embedding model id. The repo passes
namespace="exercise", which is a label, not an identity — switch models behind a shared namespace and the cache happily serves you vectors from the old one. That is exactly the failure the next section is about, with a cache added to hide it.
The rule you can never break
Everything in this part has been leading here.
Two distinct mechanisms make it break, and they fail very differently:
| Failure | Cause | Symptom |
|---|---|---|
| Dimension mismatch | 1536-dim query vector vs 768-dim stored vectors | Loud. The math does not apply; you get an exception. |
| Same dimension, different model | e.g. ada-002 (1536) at index time, 3-small (1536) at query time | Silent. Nothing errors. The two models don't share a coordinate system, so distances are meaningless and you get plausible nonsense. |
The second row is the dangerous one, and it is the one nobody tests for. Both models produce 1536 floats. Both calls succeed. The search returns the requested k documents. The LLM writes a fluent answer. The only thing wrong is that the answer has nothing to do with the question, and there is no exception anywhere in the trace. This is invariant 4 — failures are silent — in its purest form.
00:57:25 — "If you switch models, your vectors won't be comparable. Therefore, this search will fail."
That warning is stated three separate times inside one thirteen-minute chapter (00:55:23, 00:56:23, 01:00:26). It is the most repeated sentence of the hour.
And the asymmetry that makes it counter-intuitive: the generation model is completely swappable. "GPT-4, GPT-5, GPT-10, Claude, doesn't matter" 00:56:23. You can change your LLM on a Tuesday afternoon with an import and a model string. The embedding model cannot be changed without re-embedding every chunk you own. Engineers who have internalised "models are swappable" from the chat side carry that habit into the embedding side and destroy an index.
The mechanical enforcement is one line of code style. Both vector_stores.py and rag_pipeline.py declare the model exactly once, at module level, and never inline a model string at a call site:
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py:11
# (identical line at rag_pipeline.py:22)
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
That single assignment is then passed to Chroma.from_documents(...) at index time and reused by every query path in the module. If a writer wants one code takeaway from chapter 6, it is: declare the embedding model once, as a module-level constant, and never inline the model string at a call site. In a real system, promote it further — an env var plus an assertion at startup that the collection's recorded model id matches the configured one.
The three production rules that follow from it
Straight off the slide at 00:59:26:
- Use the same embedding model everywhere. Indexing and querying, identical models,
version included.
- Embedding quality > quantity. "Better embeddings on fewer documents beats worse
embeddings on more documents." A direct argument against the instinct to fix bad retrieval by dumping more data into the index.
- Test your retrieval separately. Before blaming the LLM, print what documents came
back.
That third rule comes with the number that justifies it:
00:59:26 — "90% of RAG failures are retrieval failures, not generation failures."
Like the "90% of RAG systems fail in production" figure from Part 1, this is an uncited field claim — no study, no dataset, no methodology is given. Treat the number as rhetorical and the ordering as correct: when a RAG answer is wrong, inspect the retrieved chunks before you touch the prompt or the model. And there is a ceiling on model progress that no future release removes:
00:59:26 — "If you retrieve the wrong documents, even GPT 100 can't save you."
The good path and the bad path
The worked example is deliberately mundane. Query: "What's the refund policy?"
The good path 00:58:25: the query vector captures the meaning of "refund policy"; vector search finds chunks about returns, refunds, money back — including ones with zero lexical overlap with the query — because "it's all about the actual meaning, not just the keyword"; the relevant text is retrieved; the LLM gets real context; the user gets an accurate answer. That "money back" ↔ "refund policy" match is the entire value proposition of semantic search over keyword search.
The bad path 00:58:25–00:59:26: the query vector doesn't capture meaning well; search returns random chunks; the LLM gets irrelevant context; the user gets a wrong answer or a hallucination.
Note also what this reframing does to hallucination. In chapter 6 it is not a model defect; it is the downstream symptom of bad retrieval. "The LLM can only work with what you give it" 00:59:26. That is what makes rule 3 actionable — hallucination becomes a retrieval bug you can reproduce and measure, rather than a mysterious property of the model.
And the causal chain that ties this part back to Part 3 (invariant 1 — quality is inherited forward and never recovered downstream):
00:57:25 — "Bad chunking is going to lead into bad embeddings which is going to lead into broken RAG."
00:59:26 — "Garbage embeddings equals garbage retrieval which equals garbage answers."
The embedding model has no judgement. It faithfully encodes whatever you hand it. If you hand it half of an OAuth section, you get a perfect, confident, well-formed vector for half of an OAuth section.
The Chroma workflow diagram — the reference model for any vector database
Chapter 7 spends its first seven minutes 01:01:06–01:08:15 on a single diagram, and the instructor is explicit that this is the part to remember:
01:07:13 — "this is the lecture … you should always go back to … to understand any database"
There are no screenshots for this chapter (the course's index lists 01:01:05 under "chapters with no screenshot"), so here is the structure in text.
Two layers. The app layer on top holds exactly three visible things: the incoming query, the LLM context window, and the answer going back out 01:02:08. Nothing about vectors lives up here. The Chroma DB layer underneath "stores and manages embeddings" — documents, already chunked, alongside their vectors.
The mandatory translation step in the middle. Between the two sits "gen embedding". You cannot interact with a vector database at all until the query has been transformed into a vector, because the embedding is "the format, the language that it's needed for" the database 01:03:08. The query is never sent as text.
The rest of the diagram's claims, worth having in one list:
- Storage is instrumental, never the goal. 01:02:08 — "You don't save data just for
the sake of saving." The stored material exists to be pushed through an LLM's context window with the user's query.
- "Documents" is a stand-in for any data type 01:02:08, including unstructured data.
Plain text is used because it demos cleanly.
- Queries are usually text but the modality is not fixed — "just a text or voice or
whatever" 01:03:08. It all collapses to a vector anyway.
document's content in vector form.
- Retrieval is by vector distance, not keyword match 01:04:09.
- A stored record carries more than the vector. Documents live in the store "in a form of
embeddings which are vectors which could contain a little bit more information than just the actual vectors" 01:01:06. Concretely: {id, embedding, document text, metadata}. The hands-on demo confirms this — the query result comes back with ids, documents and distances.
- The LLM does three things with the retrieved material 01:04:09: fetches related
information, interprets the query's intent, and generates new text from learned patterns. It is not merely a formatter — it can expand on retrieved context, which is exactly why Part 1's anti-hallucination prompt pattern is not optional.
- The answer is context *plus* parametric knowledge, not context alone. 01:07:13 —
the model "because it has its knowledge" is what knows how to process the retrieved material. That is the mechanism behind the previous bullet, and it is the reason a wrong chunk does not produce an obviously wrong answer: the model has enough of its own priors to smooth over the gap and hand you something fluent.
some context provided by the Chroma database." Stated as non-negotiable.
- Two possible paths after embedding the query 01:05:12: compare against stored
embeddings to find relevant documents (normal RAG), or go straight to the LLM (the degenerate no-retrieval path — worth naming, because Part 12's gate makes that path deliberate rather than accidental).
- Why this beats keyword search 01:06:12: the system can "understand and process
complex queries more accurately than traditional keyword-based search systems", producing responses that are contextually relevant and semantically rich. Hold this claim loosely — Part 6 shows exactly where pure vector search loses to keyword search (product codes, IDs, exact strings), which is why hybrid retrieval exists.
The generalisation is the point: "we're just using Chroma here in this case but any vector database at the end of the day needs to interact with the real world" 01:07:13. Swap Chroma for pgvector, Pinecone, Qdrant or FAISS and the diagram is unchanged. Only the client API moves.
Build it once by hand: the raw chromadb client
Now the mechanics, with no framework in the way. Chapter 7 is the first hands-on chapter of the course: empty folder, one file, app.py.
Versions for the record, since none of them are spoken aloud: the course repo pins requires-python = ">=3.12", ships chromadb 1.4.1 in its lockfile, and declares langchain-chroma>=1.1.0 — which is the wrapper chapter 8 onward depends on, and which Part 2 already warned you to install instead of langchain-community for the vector store.
Environment, both platforms, exactly as narrated 01:09:16–01:10:18:
# macOS / Linux
python3 -m venv venv
source venv/bin/activate # confirm the venv is ACTIVE before continuing
pip install chromadb
# Windows
python -m venv myenv # or: py -m venv myenv
myenv\Scripts\activate
pip install chromadbHe catches himself mid-command about to install into the global environment — "before I even do that… I need to create a virtual environment" 01:09:16 — for the obvious reason:
01:09:16 — "so that you don't install all these different dependencies and packages all over the place"
Then the file itself. This is the reconstruction from the narration (app.py, ~15 lines):
# app.py — reconstructed from the ch.7 narration [01:11:18]–[01:16:27].
# NOT a checked-in repo file. Raw chromadb client — no LangChain, no API key.
import chromadb
chroma_client = chromadb.Client()
collection_name = "test_collection"
collection = chroma_client.get_or_create_collection(name=collection_name)
documents = [
{"id": "doc1", "text": "hello world"},
{"id": "doc2", "text": "how are you today"},
{"id": "doc3", "text": "..."}, # third doc's text is never read aloud
]
query_text = "hello world"
for doc in documents:
collection.upsert(ids=[doc["id"]], documents=[doc["text"]])
results = collection.query(query_texts=[query_text], n_results=3)
print(results)
Small file, five real lessons.
1. A collection is a table. 01:13:22 — the clearest analogy in the chapter. test_collection is the named container that holds documents, ids and embeddings. If you know SQL, the mental mapping is: client → connection, collection → table, document → row, id → primary key.
2. get_or_create_collection, not create_collection. The vendor docs teach create_collection; he deliberately swaps it 01:11:18–01:12:18 because scripts get re-run constantly during development and create_collection conflicts when the name already exists. *(A correction to the narration: he says get-or-create "is going to go ahead rewrite everything", which is misleading. It does not wipe or overwrite anything — it returns the existing collection. The overwrite-safety comes from upsert, not from this call.)*
3. upsert, never add. This is the headline warning of the chapter 01:13:22. collection.add() has no id-collision protection, so every re-run of your script silently appends another copy of your entire corpus. upsert inserts-or-updates by id.
4. Chroma's API is list-typed everywhere. documents=[...], ids=[...], query_texts=[...] — singular-sounding operations still want lists. He learns this live: the run fails with "got an unexpected keyword argument text" because he wrote text=…; the parameter is documents, and it takes a list 01:16:27.
The generalisable move, worth more than the specific fix: when a Python SDK rejects a kwarg, don't guess the name — open the API reference for that exact method and copy the parameter names off it.
And once you have noticed that the parameters are lists, notice what the loop is doing: it calls upsert once per document with a one-element list each time. Three documents, three round-trips, three embedding batches. The same call accepts the whole corpus — collection.upsert(ids=[d["id"] for d in documents], documents=[d["text"] for d in documents]) — in one shot. Harmless at three documents; at ten thousand it is the difference between an ingest that takes a minute and one that takes an afternoon. Batch your writes.
5. The result payload is one dict of parallel lists. collection.query(query_texts=[q], n_results=3) returns ids, documents and distances as three aligned arrays 01:16:27 — not a list of record objects. n_results is the count you want back ("we want to bring in three. We can say 2, 1, 3 and so forth"). There is also a query_embeddings parameter for passing pre-computed vectors instead of text; it is mentioned once and never used anywhere in the course.
And then the observation this whole part builds toward. The query is "hello world" and doc1's text is "hello world", so doc1 comes back with a distance of 0:
01:17:29 — "So the closer to zero it is, the most similar those two are."
The other distances he reads off the output are around 0.01 and up. Flagging honestly: that 0.01 is ambiguous in the auto-generated transcript — it may be 0.0 read digit-by-digit followed by a separate number. Do not quote it as exact. The 0 for the exact match is unambiguous.
Small stuff from the hands-on chapter that still costs people an hour
- **Squiggly
import chromadbafter a successful install = wrong interpreter, not a broken
package.** 01:11:18 Point VS Code at the Python inside your venv. Do not reinstall.
- Read the vendor docs first, and know their shape. 01:08:15 He deliberately opens
Chroma's "Getting Started" page and copies snippets from it — then edits two of them (create_collection → get_or_create_collection, add → upsert) because the docs optimise for a first run and you are optimising for the hundredth.
- Scope discipline. 01:08:15 — "Obviously, this is not a Chroma DB course." The
workflow diagram is what transfers between vector databases; the method names are not.
instead of python3) and activation (Scripts\activate instead of bin/activate).
- A gap in the recorded material. Chapter 8 says the store is built with a temp directory
"as we did before" — but persist_directory never appears on screen in chapter 7, which used the bare in-memory Client(). If you follow along and feel you missed a step, you didn't; the step was never recorded.
Build it again through LangChain
Chapter 8 opens on the same database one abstraction layer up:
01:17:50 — "you can see it's really easy when we use Chroma wrapper and LangChain."
Do not let the continuity fool you into thinking the APIs are the same. They are not:
| Concept | Raw chromadb (ch. 7) | langchain_chroma (ch. 8+) |
|---|---|---|
| Create + populate | collection.upsert(ids=[…], documents=[…]) | Chroma.from_documents(documents=…, embedding=…, persist_directory=…) |
| The unit of data | {"id": …, "text": …} dict | Document(page_content=…, metadata={…}) |
| Search | collection.query(query_texts=[q], n_results=N) | vectorstore.similarity_search(q, k=N) |
| Scored search | distances in the result dict | similarity_search_with_score(q, k=N) → tuples |
| Who embeds | Chroma's bundled default model | the embedding= object you pass in |
That last row is the important one. Under LangChain you supply the embedding model explicitly, which is what puts you back in compliance with The Rule.
The corpus for every remaining demo in this part is an 8-document fixture. It is worth copying verbatim into your own scratch work, because the topic values are engineered to make the filtering demo produce an interesting result.
SAMPLE_DOCS — the 8-document fixture (vector_stores.py:14-47)
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py, lines 1-47, verbatim
from langchain_openai.embeddings import OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
import tempfile
import shutil
from dotenv import load_dotenv
load_dotenv()
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
# Sample documents
SAMPLE_DOCS = [
Document(
page_content="LangChain is a framework for developing applications powered by language models.",
metadata={"source": "langchain_docs", "topic": "overview"},
),
Document(
page_content="LangGraph is a library for building stateful, multi-actor applications with LLMs.",
metadata={"source": "langgraph_docs", "topic": "overview"},
),
Document(
page_content="Vector stores are databases optimized for storing and searching embeddings.",
metadata={"source": "vector_guide", "topic": "database"},
),
Document(
page_content="RAG combines retrieval with generation for more accurate LLM responses.",
metadata={"source": "rag_guide", "topic": "architecture"},
),
Document(
page_content="Embeddings convert text into numerical vectors for semantic similarity.",
metadata={"source": "embeddings_guide", "topic": "fundamentals"},
),
Document(
page_content="Chroma is an open-source embedding database for AI applications.",
metadata={"source": "chroma_docs", "topic": "database"},
),
Document(
page_content="FAISS is a library for efficient similarity search developed by Facebook.",
metadata={"source": "faiss_docs", "topic": "database"},
),
Document(
page_content="Pinecone is a managed vector database service for production workloads.",
metadata={"source": "pinecone_docs", "topic": "database"},
),
]
Topic distribution — memorise it, every later number depends on it: 4 × database (vector_guide, chroma_docs, faiss_docs, pinecone_docs) · 2 × overview · 1 × architecture · 1 × fundamentals.
Three notes on those first nine lines. The import is langchain_chroma, not langchain_community — Part 2 flagged that as an install-time trap and this is the file it breaks. load_dotenv() runs on line 9, before line 11 constructs the embeddings client, which is the only ordering that works because the SDK reads OPENAI_API_KEY out of the process environment at construction time. And shutil on line 6 is imported and never used; it is dead weight, not a hint that something cleans up after itself.
The baseline build-and-query is three lines of real work:
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py:50-68
def chroma_basics():
with tempfile.TemporaryDirectory() as tmpdir:
# create vector store from documents
vectorstore = Chroma.from_documents(
documents=SAMPLE_DOCS, embedding=embeddings_model, persist_directory=tmpdir
)
print(
f"Vector store created {vectorstore._collection.count()} documents and persisted."
)
# perform similarity search
query = "What is LangChain?"
results = vectorstore.similarity_search(query, k=2)
print(f"Top 2 results for query '{query}':")
for i, doc in enumerate(results):
print(
f"Result {i+1}: {doc.page_content} (Source: {doc.metadata['source']})"
)
Note the printing habit, which recurs in every single demo function in the file: the source metadata is appended to every result. He never comments on it. Adopt it anyway.
Three warnings about the scaffolding that the chapter waves off as boilerplate — "so nothing really new here" 01:17:50:
tempfile.TemporaryDirectory()means every run re-embeds the whole corpus. Eight toy
documents, so it is invisible here. At real corpus size it is an API bill and a latency wall on every execution. Use a stable persist_directory the moment your corpus stops being a toy.
vectorstore._collection.count()reaches into a private attribute. It works with
langchain-chroma 1.1.0 but it is not public API. Prefer len(vectorstore.get()["ids"]) in code you intend to keep.
- The
__main__block ships with every call commented out. That is the course's whole
execution model for these scratch files: uncomment one function, run, re-comment 01:18:52. If you clone the repo and nothing happens, that is why.
Persistence, when you are ready for it, is the same object with a real path — and one genuinely nasty naming inconsistency:
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py, lines 153-179.
# Two print statements (lines 164 and 179) elided; everything else is verbatim.
def persist_chroma():
persist_dir = "./chroma_db/"
vectorstore = Chroma.from_documents(
documents=SAMPLE_DOCS,
embedding=embeddings_model,
persist_directory=persist_dir,
)
original_count = vectorstore._collection.count()
print(f"Persisted vector store with {original_count} documents.")
# simulate restart - load from disk
del vectorstore
reloaded = Chroma(
embedding_function=embeddings_model,
persist_directory=persist_dir,
)
reloaded_count = reloaded._collection.count()
print(f"Reloaded vector store with {reloaded_count} documents.")
# verify search still works
results = reloaded.similarity_search("LangChain", k=2)
The last three lines are the whole point of the function: the reload passes no documents, only the same directory and the same embedding model, and similarity_search still works. That is what "persistent" buys you — the vectors were on disk, not in the process.
Reading the scores: the distance-vs-similarity trap
Chapter 8 is six minutes long and exists almost entirely for this one correction.
The scored variant of the search takes the same arguments and returns a different type:
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py:71-87
def similarity_search_with_scores():
with tempfile.TemporaryDirectory() as tmpdir:
# create vector store from documents
vectorstore = Chroma.from_documents(
documents=SAMPLE_DOCS, embedding=embeddings_model, persist_directory=tmpdir
)
# perform similarity search with scores
query = "Explain vector stores."
results_with_scores = vectorstore.similarity_search_with_score(query, k=3)
print(f"Top 3 results with scores for query '{query}':")
for i, (doc, score) in enumerate(results_with_scores):
final_score = 1 / (1 + score) # Convert distance to similarity
print(
f"Result {i+1}: {doc.page_content} (Score: {final_score:.4f}, Source: {doc.metadata['source']})"
)
similarity_search returns List[Document]. similarity_search_with_score returns List[Tuple[Document, float]]. That is why the loop header changes from for i, doc in enumerate(results) to for i, (doc, score) in enumerate(results_with_scores). Reusing the unscored loop body over the scored results raises AttributeError on a tuple — the single most common way this API breaks for people. (Note the method name is singular, ..._with_score, while the demo function wrapping it is plural, ..._with_scores. Easy typo.) The method also takes a where_document= kwarg alongside filter= — the same idea applied to chunk contents rather than metadata, e.g. {"$contains": "OAuth"}. The course never uses it; it is occasionally the cheapest way to force an exact-string constraint that vector search alone will not honour.
Now the numbers. Query "Explain vector stores.", k=3, and the run produces:
| Rank | Document | Raw distance | After 1/(1+d) |
|---|---|---|---|
| 1 (best) | the vector-stores doc (vector_guide) | 0.6612 | ≈ 0.6019 |
| 2 | Pinecone (pinecone_docs) | not spoken | — |
| 3 (worst) | the embeddings doc (embeddings_guide) | 1.34 | ≈ 0.4274 |
And the objection the instructor voices on the student's behalf 01:18:52: why is 0.66 a better score than something above 1?
Distance and similarity are two opposite polarities over the same geometry:
| Distance | Similarity | |
|---|---|---|
| Perfect match | 0 | 1 |
| Direction of "better" | down | up |
| Range | 0 → unbounded | roughly (0, 1] |
| Threshold shape | if score < T | if score > T |
| Sort | ascending | reverse=True |
01:19:53 — "If these were similarity scores, then ones would be very similar, right?"
The conversion. If you want a normalised, higher-is-better number, one line does it 01:20:55:
final_score = 1 / (1 + score) # distance -> similarity, vector_stores.py:84
It maps distance 0 → 1.0, is monotone decreasing, pushes large distances toward 0, and preserves the ordering exactly. Distance 0.6612 → ≈0.6019; distance 1.34 → ≈0.4274.
Polarity is per-store, and thresholds do not port. "Some vector stores actually do calculate the similarity distance" 01:19:53 — i.e. some backends hand you a similarity directly. Moving from Chroma to pgvector, Pinecone or FAISS can invert the comparison operator you need. So the practical procedure for any score you have never seen before:
- Look at the number attached to the top-ranked result.
- Smallest number at the top → you have distances; lower is better.
- Largest number at the top, trending toward 1 → you have similarities; higher is better.
- Confirm it against the method's own docstring, which is usually explicit. LangChain's
Chroma.similarity_search_with_score says, in as many words, "distance in float for each. Lower score represents more similarity."
- Need a normalised 0–1 relevance? Apply
1/(1+distance), or set the collection's space and
use similarity_search_with_relevance_scores.
- Re-run steps 1–4 every time you change vector store. The answer is not portable.
This is the same problem that Part 6's hybrid search runs into at industrial scale: BM25 scores and vector scores live on incomparable scales, which is exactly why reciprocal rank fusion throws the scores away and fuses ranks instead.
The scored variant is also a measurement instrument, not just a retrieval call. This is the highest-value use of it and the chapter never mentions it. Because the score is a number, you can hold the query fixed, build two stores that differ in exactly one thing — chunker, dimension, contextualisation strategy — and compare k=1 scores across them. That is precisely how the advanced-retrieval code later in the course evaluates itself: fcc-production-rag-part-6/02_contextual_retrieval.py calls similarity_search_with_score(query, k=1) against an original store and a contextualised store and prints both scores side by side. Rule 3 ("test your retrieval separately") stops being a slogan the moment you have that harness.
Metadata filtering: precision for free
At 01:20:55 the chapter pivots — "let's go and look at the metadata filtering because it's another important concept to have in our arsenal." Treat this as the co-equal second half of the chapter, not an appendix. It is the cheapest precision lever in the whole guide: no extra model call, no extra latency of consequence, one extra kwarg.
The demo is built as a controlled A/B — identical query, identical k, one variable changed:
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py:90-116
def metadata_filtering():
with tempfile.TemporaryDirectory() as tmpdir:
# create vector store from documents
vectorstore = Chroma.from_documents(
documents=SAMPLE_DOCS, embedding=embeddings_model, persist_directory=tmpdir
)
query = "What databases are available?"
# without metadata filtering
results = vectorstore.similarity_search(query, k=5)
print(f"Results without metadata filtering for query '{query}':")
for i, doc in enumerate(results):
print(
f"Result {i+1}: {doc.page_content} (Source: {doc.metadata['source']})"
)
# with metadata filtering
filter_criteria = {"topic": "database"}
filtered_results = vectorstore.similarity_search(
query, k=5, filter=filter_criteria
)
print(f"\nResults with metadata filtering for query '{query}':")
for i, doc in enumerate(filtered_results):
print(
f"Result {i+1}: {doc.page_content} (Source: {doc.metadata['source']})"
)
Same method, one extra kwarg. You do not call a different function 01:21:56 — it is still similarity_search, plus filter=. The filter criteria are "an object or a dictionary": keys are metadata field names attached at ingestion, values are the required match.
The result, and the headline finding:
| Run | k requested | Results returned | What came back |
|---|---|---|---|
| Unfiltered | 5 | 5 | 4 database docs + the RAG doc (topic: architecture) |
filter={"topic": "database"} | 5 | 4 | vector_guide, chroma_docs, faiss_docs, pinecone_docs |
Four, because exactly four of the eight SAMPLE_DOCS carry topic: "database" 01:22:58. The filter is a per-document predicate evaluated as a gate on the candidate set — he walks the list out loud at 01:23:59: this one is database, added; this one is database, added; this one is fundamentals, skipped; this one is architecture, skipped.
What the filter removed is the whole lesson. Unfiltered, slot #5 went to the RAG document — "RAG combines retrieval with generation for more accurate LLM responses", tagged topic: architecture. It is semantically near "What databases are available?" — it is about the same technology stack, it shares vocabulary — and pure vector similarity happily surfaced it. It does not answer the question.
Semantic proximity is not topical correctness. The embedding says "close". Your user asked "which". Those are different questions, and the vector store cannot tell them apart.
That gap is the seed of everything in Part 6: reranking, hybrid retrieval and query rewriting all exist because the nearest neighbour in embedding space is routinely not the right answer. Metadata filtering is the cheapest of those fixes and the one to reach for first — but only if you did the work earlier.
One more layer up: the retriever
vector_stores.py contains one more function that chapter 8 does not narrate but which is the bridge to Part 5, so take it as a forward pointer rather than a lesson:
# /home/oriol/lab/rag/code/production-course-main-code/vector_stores.py — the two retriever
# constructions from as_retriever(), lines 129-133 and 141-145. Print loops elided.
# plain similarity retriever
retriever = vectorstore.as_retriever(
search_type="similarity", search_kwargs={"k": 3}
)
docs = retriever.invoke("How do I build AI applications?")
# MMR retriever — diversity instead of pure proximity
mmr_retriever = vectorstore.as_retriever(
search_type="mmr",
search_kwargs={"k": 3, "fetch_k": 5}, # fetch 5 docs and return 3 diverse
)
mmr_docs = mmr_retriever.invoke("vector databases and embeddings")
as_retriever() wraps the store in the thin façade that Part 1 named — the object an LCEL chain composes with. k and filter reappear inside search_kwargs. search_type="mmr" (maximal marginal relevance) fetches fetch_k candidates and returns k diverse ones, which is the standard answer to "my top 5 results are five near-copies of the same paragraph". Realistic tuning is fetch_k ≈ 4–10× k; fetch_k=5 over a corpus of 8 documents leaves almost no room for diversity, so the shipped demo cannot really show the effect.
Checklist and decisions to make
Decisions you must make before you embed anything
| Decision | Default | Change it when |
|---|---|---|
| Embedding model | text-embedding-3-small | measured retrieval quality says otherwise |
| Dimensions | 1536 (band: 768–1536) | 3072 for accuracy at 6.5× price; 384 for footprint |
| Where the model string lives | one module-level constant | never — this is not negotiable |
| Store | Chroma with a real persist_directory | Part 10 covers pgvector/Supabase at scale |
| Metadata schema | source + topic minimum | add tenant/access/date/version before ingest |
| Distance space | Chroma's default l2 | pick cosine at collection-creation time or never |
| Score polarity | verify empirically per store | every time you change backend |
| Idempotency | explicit ids= / upsert | never rely on add/from_documents to dedupe |
Two of those rows are irreversible without a re-index: the embedding model and the distance space. One more is irreversible without a re-ingest: the metadata schema. Spend your thinking time there and treat the rest as tunable.
Pre-flight checklist
- The embedding model is declared once, at module level, and read by both the index path
and the query path. No inline model strings anywhere.
- The model id and version are recorded inside the collection, and startup asserts that
the configured model matches the one the collection was built with. langchain-chroma gives you the slot for free — collection_metadata= is written onto the collection at creation — so there is no excuse for an index whose provenance is unknowable:
EMBEDDING_MODEL = "text-embedding-3-small"
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=OpenAIEmbeddings(model=EMBEDDING_MODEL),
persist_directory=PERSIST_DIR,
collection_metadata={"embedding_model": EMBEDDING_MODEL},
)
On every reopen, read that key back — vectorstore._collection.metadata["embedding_model"], private but the only handle available — and fail loudly if it disagrees with EMBEDDING_MODEL. A crash on boot is the cheapest possible version of this bug; the alternative is discovering it six months later from a support ticket about "weird answers".
- Dimension is in the 768–1536 band unless you have a measurement justifying otherwise.
- You know whether your store returns distance or similarity, because you ran one
query and looked at the top-ranked result's number — not because you assumed. Do it once, in a test, and let the test fail if a dependency upgrade flips it.
- No threshold in your codebase compares a score without a comment naming the polarity and
the store it was calibrated against.
- Every result print includes
doc.metadata['source']. - Every document carries every metadata field anyone might ever filter on — including the
security ones. You cannot add them later without re-indexing.
- Downstream code handles
0 ≤ len(results) ≤ k, never== k. - Ingestion is idempotent: explicit ids,
upsertsemantics, and a re-run that does not
double your corpus.
- The store is persistent. No
TemporaryDirectoryon any path you run more than once. - Writes are batched. One
upsert/add_documentscall per corpus slice, not one per
document.
- Query-time embedding cost is understood as per-request and recurring, and there is a
plan (Part 9) for caching it — with the model id as the cache namespace.
- Before you touch the prompt or swap the LLM, you have printed the retrieved chunks. Rule
3, every time.
01:00:26 — "Embedding quality equals RAG quality. Get this right or nothing else matters."
Part 5 takes the store you just built, wraps it in a retriever, hangs an LCEL chain off it, and gets a real answer out — and then teaches you how to find out why that answer is wrong.