AI interview rounds · Lesson 5 of 8 · 25 min
The RAG design round
Part 1Basics
RAG stands for retrieval-augmented generation. A model doesn't know your documents, so you find the relevant passages and put them in the prompt. It's like an open-book exam: the student's quality matters, but if you hand over the wrong book, nothing else helps.
That gives you the key idea for this round: measure retrieval and answers separately. When an answer is bad, you need to know which part failed. One reported rejection pattern is a candidate who can't state retrieval quality as a number.
Words you need
- Chunk
- A passage cut from a document. Search returns chunks, not whole files.
- Hybrid search
- Combining keyword search, which catches exact terms, with vector search, which catches meaning.
- Rerank
- A slower, more accurate second pass over the top results.
- Recall@k
- Of the passages that matter, the share that reached the top k.
- Faithfulness
- Whether the answer is supported by the retrieved text.
A worked example
The prompt: 'Design a question-answering system for our 200-page internal policy documents.' Meera asks three things first: who uses it, how often the documents change, and what a wrong answer costs. Wrong answers about leave policy are expensive, so she designs for citations and for saying 'I don't know'. Then she draws it from the question backwards, and states her target up front: recall@5 above 0.9 on 50 labelled questions before she'd tune the prompt at all.
Part 2The question
How do you design a system that answers questions from your own documents?
By the end of this lesson, you should be able to:
- Draw the indexing and query paths
- Choose chunking and search with reasons
- Measure retrieval and answers separately
- Handle permissions and stale documents
Part 3In detail
Step by step
- Clarify. Users, document size and change rate, permissions, cost of a wrong answer, and latency.
- Indexing path. Parse, chunk, embed, and store with metadata, including who may read each chunk.
- Query path. Rewrite the query if needed, run hybrid search filtered by permission, rerank, then build the prompt.
- Generate with citations. Ask for citations, check them in code, and allow 'I don't know'.
- Measure. A labelled question set: recall@k and MRR for retrieval, faithfulness and correctness for answers.
- Handle change and failure. Re-index on change, track what's stale, and have a fallback when search returns little.
Do you need RAG at all?
One rejection pattern reported is treating retrieval as the default. A small, stable document can simply go in the prompt. A question about live data may need a database query through a tool. Say why you're choosing retrieval.
Permissions go in the search
Filter by access at search time. If you filter after generating, a user's answer has already been shaped by a document they shouldn't have seen. Interviewers like this point.
Symptom to cause
| You see | Check first | Usual fix |
|---|---|---|
| Misses exact terms like error codes | Keyword search | Add hybrid search |
| Finds the right document, wrong passage | Chunk size and rerank | Change the chunking, and add a reranker |
| Right passage, wrong answer | Prompt and context order | Tighten the prompt, and show less text |
| Invented citations | Citation check | Validate citations in code |
| Old answers | Index freshness | Re-index on change, and add dates to metadata |
The code, explained
This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.
Splitting documents
Search finds chunks rather than whole documents. Too big blurs the meaning, and too small loses the context.
1/** Overlap means a sentence cut at a boundary still appears whole in the next chunk. */2export function chunkWords(text: string, size: number, overlap: number): string[] {3 if (overlap >= size) throw new Error("overlap must be smaller than size");4 const words = text.split(/\s+/).filter(Boolean);5 const out: string[] = [];6 for (let i = 0; i < words.length; i += size - overlap) {7 out.push(words.slice(i, i + size).join(" "));8 if (i + size >= words.length) break;9 }10 return out;11}i += size - overlap- Each chunk starts a little before the last one ended.
overlap >= size- This would never move forward, so it's refused up front.
i + size >= words.length- Stops once the last word is covered, so no tiny leftover chunk appears.
Tested by npm run test:modules.
Cosine similarity
The basic measure for comparing two embeddings.
1/** Compares direction, not length, so a long document doesn't win by being long. */2export function cosine(a: number[], b: number[]): number {3 let dot = 0;4 let na = 0;5 let nb = 0;6 for (let i = 0; i < a.length; i++) {7 dot += a[i] * b[i];8 na += a[i] * a[i];9 nb += b[i] * b[i];10 }11 return na && nb ? dot / Math.sqrt(na * nb) : 0;12}dot / Math.sqrt(na * nb)- Dividing by both lengths leaves only the angle between the vectors.
na && nb- An all-zero vector has no direction, so it returns 0 instead of NaN.
Tested by npm run test:modules.
Hybrid search with rank fusion
Keyword search finds exact terms like error codes, and vectors find meaning. You usually want both.
1/** Keyword and vector scores use different scales, so merge by rank instead of by score. */2export function reciprocalRankFusion(rankings: string[][], k = 60): string[] {3 const scores = new Map<string, number>();4 for (const ranking of rankings) {5 ranking.forEach((id, rank) => scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank + 1)));6 }7 return [...scores].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])).map(([id]) => id);8}1 / (k + rank + 1)- Uses only position, so the two scoring scales never have to be compared.
k = 60- The constant commonly used for this method. It stops first place from dominating.
a[0].localeCompare(b[0])- Ties break by id, so the result is repeatable.
Tested by npm run test:modules.
Measuring retrieval
Measure this before you look at answers. If the right text never arrives, no prompt can help.
1export interface Labelled {2 retrieved: string[];3 relevant: string[];4}5 6/** Of the documents that matter, what share reached the top k? Averaged over questions. */7export function recallAtK(rows: Labelled[], k: number): number {8 const each = rows.map((r) => r.relevant.filter((id) => r.retrieved.slice(0, k).includes(id)).length / r.relevant.length);9 return each.reduce((a, b) => a + b, 0) / rows.length;10}11 12/** How high does the first useful document sit? 1 means first place. */13export function meanReciprocalRank(rows: Labelled[]): number {14 const each = rows.map((r) => {15 const at = r.retrieved.findIndex((id) => r.relevant.includes(id));16 return at === -1 ? 0 : 1 / (at + 1);17 });18 return each.reduce((a, b) => a + b, 0) / rows.length;19}r.retrieved.slice(0, k)- Only what actually reaches the prompt counts.
r.relevant.length- Recall needs labelled answers, so you need a labelled set.
1 / (at + 1)- First place scores 1, second 0.5, and a miss 0.
Tested by npm run test:modules.
What can go wrong
- Tuning prompts before measuring retrieval
- If the right text never arrives, you're polishing the wrong layer.
- Testing on questions you wrote from the chunks
- That flatters recall. Use real user questions if you can.
- Forgetting permissions
- An internal search that leaks HR documents is a serious failure.
Part 4Questions and answers
Answer out loud first, then open each one.
Why merge rankings by rank, not by score?
Recall@5 is 0.95, yet answers are poor. Where do you look?
Too large or too small a chunk: what happens?
How do you handle a question the documents don't cover?
Part 5Going further
Drill: Whiteboard it in 35 minutes (35 min)
- Prompt: 'Design search and answers over 2 million support tickets.'
- Spend 5 minutes on questions, and write your assumptions.
- Draw the indexing and query paths.
- State your metrics and targets.
- Ask an AI to play interviewer and probe your weakest choice.
Score yourself
Tick a line only if you did it without prompting.
0 of 6 Repeat the drill, and aim for the lines you missed.
Keep the fundamentals fresh with the critical basics.
Who says so
Checked in November 2026. Vendors and individuals often report what candidates told them, so treat their details as reports.
- What applied-AI engineer interviews test Individual, July 2026
Next: give the model tools, and the questions get harder. The agent round.