/Learn
Deep dives · AI handbook · retrieval

Embeddings and vector search

LearnDeep diveTier 2 · MechanismsAnalyze20 minRAG and retrieval

You will be able to: explain embeddings and vector search and answer its interview questions.

Prove it: Reciprocal Rank Fusion (RRF) Hybrid Search

AI engineeringDeep dives

Study with AI

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

Embedding models turn text, images or audio into vectors, so similar meanings land close together. Approximate indexes find the nearest vectors quickly. Together they power semantic search, recommendations, deduplication and the retrieval half of RAG.

What embeddings give you
  • Search by meaning instead of exact words
  • Compare across languages with multilingual models
  • Cluster, deduplicate and recommend
  • Feed relevant context to an LLM
Where they fall short
  • Exact identifiers such as SKUs, error codes and names
  • Negation and fine-grained constraints (“not vegan”)
  • Freshness unless you re-embed on change
  • Permissions unless you filter inside the query

1.2Key concepts

Ideas this deep dive relies on, in one sentence each.

01Quantisation

Storing weights or activations at lower precision to save memory and speed up inference.

02Embedding

A vector representing meaning so similar items are close together.

03RAG

Retrieval-augmented generation: fetch relevant documents and give them to the model.

04Reranker

A model that reads query and candidate together to order results precisely.

05BM25

Keyword ranking that rewards rare terms, saturates repeats and discounts long documents.

06RRF

Reciprocal rank fusion: merges rankings using positions, not scores.

1.3Building semantic search for a service catalogue

A concrete example first, so the mechanics have something to attach to.

The wellness app has 40,000 services across 3,000 clinics. Keyword search fails on ‘something for a stiff neck after long flights’ because no listing contains those words.

A short document per service feeds the embedding: title, category, description and benefits. Postgres stores the 768-dimension vectors with pgvector and an HNSW index, alongside city, price and clinic ID for filtering.

The same model embeds each query, which then runs twice: vector nearest neighbours and BM25 keyword search, both filtered by city. Reciprocal rank fusion merges the results, and a cross-encoder reranks the top 30.

Offline evaluation on 300 labelled queries shows recall at 10 rising substantially over keyword search, and bookings from search improve in an A/B test. Recompute embeddings only when a service description changes.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. Why combine keyword and vector search?
  2. HNSW or IVF?
  3. How do you pick an embedding model?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1The retrieval stack

The parts and how they connect.

INDEXINGQUERY TIMESource recordstitle · text ·metadataPrepare textchunk · cleanEmbedding modelbatch jobHybrid indexHNSW · BM25 ·filtersQueryuser text + filtersQuery embeddersame modelSearch + fusionkNN + BM25 → RRFRerankercross-encoderupsertkNN + BM25top 50
ClientServiceData storeAI component
Boxes are components; arrows show calls and data flow.
Embedding model
Maps input to a fixed-length vector. Use the same model, version and preprocessing for documents and queries; changing models means re-embedding everything.
Similarity
Usually cosine similarity, or dot product on normalised vectors.
Approximate index
HNSW builds a navigable graph for fast, high-recall search; IVF partitions vectors into clusters; product quantisation compresses vectors to save memory at some recall cost.
Hybrid search
Keyword search catches exact terms; vector search catches meaning. Reciprocal rank fusion combines ranked lists without tuning score scales.
Reranker
A cross-encoder reads the query and each candidate together and scores relevance much more accurately, but is too slow to run on the whole corpus.

3.2Index types

NameMeaning
Flat (exact)Compare against every vector. Perfect recall; fine up to around 100k vectors.
HNSWGraph index; fast and accurate; uses more memory; supports incremental inserts.
IVFCluster-based; cheaper to build; tune the number of clusters probed.
PQ / int8 / binaryCompressed vectors; large memory savings with some recall loss.
Matryoshka embeddingsModels trained so vectors can be truncated to fewer dimensions with graceful quality loss.

3.3A hybrid query

One run, step by step.

AppSearch APIEmbedding modelIndexReranker‘stiff neck after flights’, city=Puneembed query768-d vectorkNN top 50 + BM25 top 50, city filtertwo ranked listsreciprocal rank fusionrerank top 30scored listresults
Solid arrows are calls; dashed arrows are responses.
  1. Filters run inside both searches so results never include other cities.
  2. The reranker is where most of the precision comes from.

3.4Code

Working code for the core of it.

pgvector table and index

pgvector-table-index.sqlSQL
1CREATE EXTENSION IF NOT EXISTS vector;2CREATE TABLE service_vec (3  service_id uuid PRIMARY KEY, city text NOT NULL, price int,4  model_version text NOT NULL,5  embedding vector(768) NOT NULL6);7CREATE INDEX ON service_vec USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);8-- query time9SET hnsw.ef_search = 80;10SELECT service_id FROM service_vec11WHERE city = $2 ORDER BY embedding <=> $1 LIMIT 50;
11 linesUTF-8SQL

Reciprocal rank fusion (TypeScript)

reciprocal-rank-fusion.tsTypeScript
1// Merge rankings from different retrievers (BM25, vectors, …) without calibrating their scores:2// each list contributes 1 / (k + rank) for every document it returns.3function rrf(rankings: string[][], k = 60): string[] {4  const scores = new Map<string, number>();5  for (const list of rankings) {6    list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + 1 / (k + i + 1)));7  }8  return [...scores].sort((a, b) => b[1] - a[1]).map(([id]) => id);9}
9 linesUTF-8TypeScript

Recall at k for your own test set (TypeScript)

recall-at-k.tsTypeScript
1// results: query -> ranked document ids; labels: query -> ids a person marked relevant.2function recallAtK(results: Map<string, string[]>, labels: Map<string, Set<string>>, k = 10): number {3  let total = 0;4  for (const [query, relevant] of labels) {5    const top = (results.get(query) ?? []).slice(0, k);6    total += top.filter((id) => relevant.has(id)).length / relevant.size;7  }8  return total / labels.size;9}
9 linesUTF-8TypeScript

3.5Tested code, explained

A working implementation, then what each part does and why.

Nearest-neighbour search with HNSW

Comparing against every vector is too slow. HNSW walks a graph: long jumps near the top, fine steps at the bottom.

hnsw.tsTypeScript
1type Vec = number[];2const dist = (a: Vec, b: Vec) => a.reduce((s, x, i) => s + (x - b[i]) ** 2, 0);3 4/** A compact HNSW: layers of small-world graphs, searched greedily from the top down. */5export class Hnsw {6  private vecs: Vec[] = [];7  private links: number[][][] = []; // links[node][layer] = neighbours8  private entry = -1;9  private top = -1;10  private M: number;11  private efConstruction: number;12  private rand: () => number;13 14  constructor(M = 16, efConstruction = 100, rand: () => number = Math.random) {15    this.M = M;16    this.efConstruction = efConstruction;17    this.rand = rand;18  }19 20  private searchLayer(q: Vec, start: number, ef: number, layer: number): number[] {21    const seen = new Set([start]);22    const candidates = [start];23    const best = [start];24    while (candidates.length > 0) {25      candidates.sort((a, b) => dist(q, this.vecs[a]) - dist(q, this.vecs[b]));26      const c = candidates.shift()!;27      if (best.length >= ef && dist(q, this.vecs[c]) > dist(q, this.vecs[best[best.length - 1]])) break;28      for (const n of this.links[c][layer] ?? []) {29        if (seen.has(n)) continue;30        seen.add(n);31        candidates.push(n);32        best.push(n);33        best.sort((a, b) => dist(q, this.vecs[a]) - dist(q, this.vecs[b]));34        if (best.length > ef) best.pop();35      }36    }37    return best;38  }39 40  insert(v: Vec): void {41    const id = this.vecs.length;42    const level = Math.floor(-Math.log(Math.max(this.rand(), 1e-9)) / Math.log(this.M));43    this.vecs.push(v);44    this.links.push(Array.from({ length: level + 1 }, () => []));45    if (this.entry < 0) {46      this.entry = id;47      this.top = level;48      return;49    }50    let ep = this.entry;51    for (let l = this.top; l > level; l--) ep = this.searchLayer(v, ep, 1, l)[0];52    for (let l = Math.min(this.top, level); l >= 0; l--) {53      const found = this.searchLayer(v, ep, this.efConstruction, l);54      const limit = l === 0 ? this.M * 2 : this.M;55      this.links[id][l] = found.slice(0, this.M);56      for (const n of this.links[id][l]) {57        const list = this.links[n][l];58        list.push(id);59        if (list.length > limit) {60          list.sort((a, b) => dist(this.vecs[n], this.vecs[a]) - dist(this.vecs[n], this.vecs[b]));61          list.length = limit;62        }63      }64      ep = found[0];65    }66    if (level > this.top) {67      this.entry = id;68      this.top = level;69    }70  }71 72  search(q: Vec, k: number, ef = 50): number[] {73    let ep = this.entry;74    for (let l = this.top; l > 0; l--) ep = this.searchLayer(q, ep, 1, l)[0];75    return this.searchLayer(q, ep, Math.max(ef, k), 0).slice(0, k);76  }77}
77 linesUTF-8TypeScript
-Math.log(...) / Math.log(this.M)
A random level, so few nodes reach the top layers. That creates the long jumps.
for (let l = this.top; l > 0; l--)
Descends greedily, using each layer's best match as the next layer's start.
ef
How many candidates to keep. Raising it improves recall and costs time, and it's the main dial you tune.
list.length = limit
Trims each node's links, keeping memory bounded.

Tested by npm run test:modules.

Keyword ranking with BM25

Exact terms such as error codes are where vectors do worst and BM25 does best.

bm25.tsTypeScript
1const tokenize = (s: string) => s.toLowerCase().match(/[a-z0-9]+/g) ?? [];2 3/** Okapi BM25 over an inverted index: term -> list of [doc, term frequency]. */4export class Bm25 {5  private index = new Map<string, [number, number][]>();6  private ids: string[] = [];7  private lengths: number[] = [];8  private k1: number;9  private b: number;10 11  constructor(k1 = 1.2, b = 0.75) {12    this.k1 = k1;13    this.b = b;14  }15 16  add(id: string, text: string): void {17    const doc = this.ids.push(id) - 1;18    const terms = tokenize(text);19    this.lengths.push(terms.length);20    const tf = new Map<string, number>();21    for (const t of terms) tf.set(t, (tf.get(t) ?? 0) + 1);22    for (const [t, n] of tf) {23      if (!this.index.has(t)) this.index.set(t, []);24      this.index.get(t)!.push([doc, n]);25    }26  }27 28  search(query: string, top = 10): { id: string; score: number }[] {29    const N = this.ids.length;30    const avg = this.lengths.reduce((a, b) => a + b, 0) / Math.max(N, 1);31    const scores = new Map<number, number>();32    for (const term of new Set(tokenize(query))) {33      const postings = this.index.get(term) ?? [];34      const idf = Math.log(1 + (N - postings.length + 0.5) / (postings.length + 0.5));35      for (const [doc, tf] of postings) {36        const norm = tf + this.k1 * (1 - this.b + (this.b * this.lengths[doc]) / avg);37        scores.set(doc, (scores.get(doc) ?? 0) + (idf * tf * (this.k1 + 1)) / norm);38      }39    }40    return [...scores]41      .sort((a, b) => b[1] - a[1])42      .slice(0, top)43      .map(([doc, score]) => ({ id: this.ids[doc], score }));44  }45}
45 linesUTF-8TypeScript
Math.log(1 + (N - postings.length + 0.5) / ...)
Inverse document frequency: rare terms count for more.
tf * (this.k1 + 1)
Repeats help, but saturate, so stuffing a term doesn't win.
this.lengths[doc]) / avg
Long documents get discounted, since they match more by chance.

Tested by npm run test:modules.

Merging rankings with RRF

Keyword and vector scores aren't comparable. Ranks are.

rrf.tsTypeScript
1export interface RrfOptions {2  k?: number;3  weights?: number[];4  top?: number;5}6 7/** Reciprocal rank fusion: each list adds weight / (k + rank). Raw scores never enter the sum. */8export function rrf(lists: string[][], { k = 60, weights, top }: RrfOptions = {}): { id: string; score: number }[] {9  const scores = new Map<string, number>();10  lists.forEach((list, l) =>11    list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + (weights?.[l] ?? 1) / (k + i + 1)))12  );13  const ranked = [...scores].map(([id, score]) => ({ id, score })).sort((a, b) => b.score - a.score);14  return top === undefined ? ranked : ranked.slice(0, top);15}
15 linesUTF-8TypeScript
/ (k + i + 1)
Only position counts. An item near the top of both lists beats one at the top of just one.
k = 60
Softens the gap between top ranks. 60 is the value from the original paper.

Tested by npm run test:modules.

3.6Choosing where vectors live

Inside your existing database

  • pgvector, OpenSearch k-NN or Elasticsearch: one system, transactional updates, easy filters.
  • Good into tens of millions of vectors with tuning.

Dedicated vector database

  • Pinecone, Weaviate, Qdrant, Milvus: built for very large indexes, multi-tenancy and fast filtered search.
  • Adds another system to operate and keep in sync.

On device

  • Small embedding models and a local index for offline semantic search over a user’s own data.
  • Keeps private data on the phone.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

Why combine keyword and vector search?
Vectors miss exact tokens like product codes; keywords miss paraphrases. Fusion gets both, and a reranker sorts the final list.
HNSW or IVF?
HNSW for high recall and incremental updates with enough memory; IVF with PQ when memory is tight and the index is rebuilt in batches.
How do you pick an embedding model?
Label a few hundred real queries, measure recall@k and latency for candidates, check language coverage and cost, then decide.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Models and stores

Show detailsHide details
NeedOptionsWhen to pick
Embedding modelsHosted embedding APIs · open models (e.g. BGE, E5, multilingual variants)Benchmark on your queries; check multilingual quality, dimension and cost.
Vector storespgvector · OpenSearch · Qdrant · Weaviate · Milvus · PineconeStart in the database you run; move when scale or filtering demands.
RerankersCross-encoder models · hosted rerank APIsAlmost always worth it for top-k quality.
EvaluationLabelled query sets · recall@k · nDCGWithout labels you cannot compare models or settings.

5.2Retrieval metrics

Show detailsHide details
MetricWhy it matters
recall@kDid the right items make the candidate list?
nDCG / MRRAre the best items near the top?
Query latency p95Index parameters trade speed for recall.
Index freshness lagTime from change to searchable.

5.3Pitfalls

Show detailsHide details
CaseHow it is handled
Changed embedding modelAll vectors must be re-embedded; keep model_version on every row and dual-write during migration.
Exact code searches failHybrid search with BM25 fixes it.
Filtered search returns too few resultsIncrease candidate pool or use an index with native filtering.
Long documents embedded wholeChunk by structure; long inputs blur meaning.
AI Engineer · Embeddings and vector search