Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
Embedding models turn text, images or audio into vectors, so similar meanings land close together. Approximate indexes find the nearest vectors quickly. Together they power semantic search, recommendations, deduplication and the retrieval half of RAG.
- Search by meaning instead of exact words
- Compare across languages with multilingual models
- Cluster, deduplicate and recommend
- Feed relevant context to an LLM
- Exact identifiers such as SKUs, error codes and names
- Negation and fine-grained constraints (“not vegan”)
- Freshness unless you re-embed on change
- Permissions unless you filter inside the query
1.2Key concepts
Ideas this deep dive relies on, in one sentence each.
Storing weights or activations at lower precision to save memory and speed up inference.
A vector representing meaning so similar items are close together.
Retrieval-augmented generation: fetch relevant documents and give them to the model.
A model that reads query and candidate together to order results precisely.
Keyword ranking that rewards rare terms, saturates repeats and discounts long documents.
Reciprocal rank fusion: merges rankings using positions, not scores.
1.3Building semantic search for a service catalogue
A concrete example first, so the mechanics have something to attach to.
The wellness app has 40,000 services across 3,000 clinics. Keyword search fails on ‘something for a stiff neck after long flights’ because no listing contains those words.
A short document per service feeds the embedding: title, category, description and benefits. Postgres stores the 768-dimension vectors with pgvector and an HNSW index, alongside city, price and clinic ID for filtering.
The same model embeds each query, which then runs twice: vector nearest neighbours and BM25 keyword search, both filtered by city. Reciprocal rank fusion merges the results, and a cross-encoder reranks the top 30.
Offline evaluation on 300 labelled queries shows recall at 10 rising substantially over keyword search, and bookings from search improve in an A/B test. Recompute embeddings only when a service description changes.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- Why combine keyword and vector search?
- HNSW or IVF?
- How do you pick an embedding model?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1The retrieval stack
The parts and how they connect.
- Embedding model
- Maps input to a fixed-length vector. Use the same model, version and preprocessing for documents and queries; changing models means re-embedding everything.
- Similarity
- Usually cosine similarity, or dot product on normalised vectors.
- Approximate index
- HNSW builds a navigable graph for fast, high-recall search; IVF partitions vectors into clusters; product quantisation compresses vectors to save memory at some recall cost.
- Hybrid search
- Keyword search catches exact terms; vector search catches meaning. Reciprocal rank fusion combines ranked lists without tuning score scales.
- Reranker
- A cross-encoder reads the query and each candidate together and scores relevance much more accurately, but is too slow to run on the whole corpus.
3.2Index types
| Name | Meaning |
|---|---|
| Flat (exact) | Compare against every vector. Perfect recall; fine up to around 100k vectors. |
| HNSW | Graph index; fast and accurate; uses more memory; supports incremental inserts. |
| IVF | Cluster-based; cheaper to build; tune the number of clusters probed. |
| PQ / int8 / binary | Compressed vectors; large memory savings with some recall loss. |
| Matryoshka embeddings | Models trained so vectors can be truncated to fewer dimensions with graceful quality loss. |
3.3A hybrid query
One run, step by step.
- Filters run inside both searches so results never include other cities.
- The reranker is where most of the precision comes from.
3.4Code
Working code for the core of it.
pgvector table and index
1CREATE EXTENSION IF NOT EXISTS vector;2CREATE TABLE service_vec (3 service_id uuid PRIMARY KEY, city text NOT NULL, price int,4 model_version text NOT NULL,5 embedding vector(768) NOT NULL6);7CREATE INDEX ON service_vec USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);8-- query time9SET hnsw.ef_search = 80;10SELECT service_id FROM service_vec11WHERE city = $2 ORDER BY embedding <=> $1 LIMIT 50;Reciprocal rank fusion (TypeScript)
1// Merge rankings from different retrievers (BM25, vectors, …) without calibrating their scores:2// each list contributes 1 / (k + rank) for every document it returns.3function rrf(rankings: string[][], k = 60): string[] {4 const scores = new Map<string, number>();5 for (const list of rankings) {6 list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + 1 / (k + i + 1)));7 }8 return [...scores].sort((a, b) => b[1] - a[1]).map(([id]) => id);9}Recall at k for your own test set (TypeScript)
1// results: query -> ranked document ids; labels: query -> ids a person marked relevant.2function recallAtK(results: Map<string, string[]>, labels: Map<string, Set<string>>, k = 10): number {3 let total = 0;4 for (const [query, relevant] of labels) {5 const top = (results.get(query) ?? []).slice(0, k);6 total += top.filter((id) => relevant.has(id)).length / relevant.size;7 }8 return total / labels.size;9}3.5Tested code, explained
A working implementation, then what each part does and why.
Nearest-neighbour search with HNSW
Comparing against every vector is too slow. HNSW walks a graph: long jumps near the top, fine steps at the bottom.
1type Vec = number[];2const dist = (a: Vec, b: Vec) => a.reduce((s, x, i) => s + (x - b[i]) ** 2, 0);3 4/** A compact HNSW: layers of small-world graphs, searched greedily from the top down. */5export class Hnsw {6 private vecs: Vec[] = [];7 private links: number[][][] = []; // links[node][layer] = neighbours8 private entry = -1;9 private top = -1;10 private M: number;11 private efConstruction: number;12 private rand: () => number;13 14 constructor(M = 16, efConstruction = 100, rand: () => number = Math.random) {15 this.M = M;16 this.efConstruction = efConstruction;17 this.rand = rand;18 }19 20 private searchLayer(q: Vec, start: number, ef: number, layer: number): number[] {21 const seen = new Set([start]);22 const candidates = [start];23 const best = [start];24 while (candidates.length > 0) {25 candidates.sort((a, b) => dist(q, this.vecs[a]) - dist(q, this.vecs[b]));26 const c = candidates.shift()!;27 if (best.length >= ef && dist(q, this.vecs[c]) > dist(q, this.vecs[best[best.length - 1]])) break;28 for (const n of this.links[c][layer] ?? []) {29 if (seen.has(n)) continue;30 seen.add(n);31 candidates.push(n);32 best.push(n);33 best.sort((a, b) => dist(q, this.vecs[a]) - dist(q, this.vecs[b]));34 if (best.length > ef) best.pop();35 }36 }37 return best;38 }39 40 insert(v: Vec): void {41 const id = this.vecs.length;42 const level = Math.floor(-Math.log(Math.max(this.rand(), 1e-9)) / Math.log(this.M));43 this.vecs.push(v);44 this.links.push(Array.from({ length: level + 1 }, () => []));45 if (this.entry < 0) {46 this.entry = id;47 this.top = level;48 return;49 }50 let ep = this.entry;51 for (let l = this.top; l > level; l--) ep = this.searchLayer(v, ep, 1, l)[0];52 for (let l = Math.min(this.top, level); l >= 0; l--) {53 const found = this.searchLayer(v, ep, this.efConstruction, l);54 const limit = l === 0 ? this.M * 2 : this.M;55 this.links[id][l] = found.slice(0, this.M);56 for (const n of this.links[id][l]) {57 const list = this.links[n][l];58 list.push(id);59 if (list.length > limit) {60 list.sort((a, b) => dist(this.vecs[n], this.vecs[a]) - dist(this.vecs[n], this.vecs[b]));61 list.length = limit;62 }63 }64 ep = found[0];65 }66 if (level > this.top) {67 this.entry = id;68 this.top = level;69 }70 }71 72 search(q: Vec, k: number, ef = 50): number[] {73 let ep = this.entry;74 for (let l = this.top; l > 0; l--) ep = this.searchLayer(q, ep, 1, l)[0];75 return this.searchLayer(q, ep, Math.max(ef, k), 0).slice(0, k);76 }77}-Math.log(...) / Math.log(this.M)- A random level, so few nodes reach the top layers. That creates the long jumps.
for (let l = this.top; l > 0; l--)- Descends greedily, using each layer's best match as the next layer's start.
ef- How many candidates to keep. Raising it improves recall and costs time, and it's the main dial you tune.
list.length = limit- Trims each node's links, keeping memory bounded.
Tested by npm run test:modules.
Keyword ranking with BM25
Exact terms such as error codes are where vectors do worst and BM25 does best.
1const tokenize = (s: string) => s.toLowerCase().match(/[a-z0-9]+/g) ?? [];2 3/** Okapi BM25 over an inverted index: term -> list of [doc, term frequency]. */4export class Bm25 {5 private index = new Map<string, [number, number][]>();6 private ids: string[] = [];7 private lengths: number[] = [];8 private k1: number;9 private b: number;10 11 constructor(k1 = 1.2, b = 0.75) {12 this.k1 = k1;13 this.b = b;14 }15 16 add(id: string, text: string): void {17 const doc = this.ids.push(id) - 1;18 const terms = tokenize(text);19 this.lengths.push(terms.length);20 const tf = new Map<string, number>();21 for (const t of terms) tf.set(t, (tf.get(t) ?? 0) + 1);22 for (const [t, n] of tf) {23 if (!this.index.has(t)) this.index.set(t, []);24 this.index.get(t)!.push([doc, n]);25 }26 }27 28 search(query: string, top = 10): { id: string; score: number }[] {29 const N = this.ids.length;30 const avg = this.lengths.reduce((a, b) => a + b, 0) / Math.max(N, 1);31 const scores = new Map<number, number>();32 for (const term of new Set(tokenize(query))) {33 const postings = this.index.get(term) ?? [];34 const idf = Math.log(1 + (N - postings.length + 0.5) / (postings.length + 0.5));35 for (const [doc, tf] of postings) {36 const norm = tf + this.k1 * (1 - this.b + (this.b * this.lengths[doc]) / avg);37 scores.set(doc, (scores.get(doc) ?? 0) + (idf * tf * (this.k1 + 1)) / norm);38 }39 }40 return [...scores]41 .sort((a, b) => b[1] - a[1])42 .slice(0, top)43 .map(([doc, score]) => ({ id: this.ids[doc], score }));44 }45}Math.log(1 + (N - postings.length + 0.5) / ...)- Inverse document frequency: rare terms count for more.
tf * (this.k1 + 1)- Repeats help, but saturate, so stuffing a term doesn't win.
this.lengths[doc]) / avg- Long documents get discounted, since they match more by chance.
Tested by npm run test:modules.
Merging rankings with RRF
Keyword and vector scores aren't comparable. Ranks are.
1export interface RrfOptions {2 k?: number;3 weights?: number[];4 top?: number;5}6 7/** Reciprocal rank fusion: each list adds weight / (k + rank). Raw scores never enter the sum. */8export function rrf(lists: string[][], { k = 60, weights, top }: RrfOptions = {}): { id: string; score: number }[] {9 const scores = new Map<string, number>();10 lists.forEach((list, l) =>11 list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + (weights?.[l] ?? 1) / (k + i + 1)))12 );13 const ranked = [...scores].map(([id, score]) => ({ id, score })).sort((a, b) => b.score - a.score);14 return top === undefined ? ranked : ranked.slice(0, top);15}/ (k + i + 1)- Only position counts. An item near the top of both lists beats one at the top of just one.
k = 60- Softens the gap between top ranks. 60 is the value from the original paper.
Tested by npm run test:modules.
3.6Choosing where vectors live
Inside your existing database
- pgvector, OpenSearch k-NN or Elasticsearch: one system, transactional updates, easy filters.
- Good into tens of millions of vectors with tuning.
Dedicated vector database
- Pinecone, Weaviate, Qdrant, Milvus: built for very large indexes, multi-tenancy and fast filtered search.
- Adds another system to operate and keep in sync.
On device
- Small embedding models and a local index for offline semantic search over a user’s own data.
- Keeps private data on the phone.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
Why combine keyword and vector search?
HNSW or IVF?
How do you pick an embedding model?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Models and stores
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| Embedding models | Hosted embedding APIs · open models (e.g. BGE, E5, multilingual variants) | Benchmark on your queries; check multilingual quality, dimension and cost. |
| Vector stores | pgvector · OpenSearch · Qdrant · Weaviate · Milvus · Pinecone | Start in the database you run; move when scale or filtering demands. |
| Rerankers | Cross-encoder models · hosted rerank APIs | Almost always worth it for top-k quality. |
| Evaluation | Labelled query sets · recall@k · nDCG | Without labels you cannot compare models or settings. |
5.2Retrieval metrics
Show detailsHide details
| Metric | Why it matters |
|---|---|
| recall@k | Did the right items make the candidate list? |
| nDCG / MRR | Are the best items near the top? |
| Query latency p95 | Index parameters trade speed for recall. |
| Index freshness lag | Time from change to searchable. |
5.3Pitfalls
Show detailsHide details
| Case | How it is handled |
|---|---|
| Changed embedding model | All vectors must be re-embedded; keep model_version on every row and dual-write during migration. |
| Exact code searches fail | Hybrid search with BM25 fixes it. |
| Filtered search returns too few results | Increase candidate pool or use an index with native filtering. |
| Long documents embedded whole | Chunk by structure; long inputs blur meaning. |