Basics
Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.
1.1Product in plain words
Who uses it, what a normal day looks like, and why it is harder than it seems.
An internal assistant for support and sales staff that answers questions using the company’s own documents (policies, manuals, tickets and wiki pages) and always shows where each answer came from.
Who uses it
- Support agents: need correct answers fast while a customer waits.
- Sales staff: need the latest pricing and policy, by region.
- Compliance: needs restricted documents to stay restricted.
A day in the life
- An agent asks: ‘What’s the refund policy for annual plans in the UAE?’
- The system searches 5 million documents, but only those she is allowed to see.
- The answer appears with numbered citations like [1] and [2].
- Hovering a citation shows the exact paragraph from the policy, with a link.
- She marks it helpful; a wrong answer would be flagged and added to the test set.
Why it is hard
Among millions of documents, retrieval quality decides answer quality.
An answer must never be built from a document the user can’t open.
Policies change; yesterday’s answer can be wrong today.
Users need to verify answers quickly, or they stop using it.
Like a librarian who first fetches the three most relevant pages from the shelves you’re allowed into, then answers using only those pages, with sticky notes showing where each fact came from.
1.2Key concepts
The building blocks this design uses. Each one appears again in the answer.
A vector that represents the meaning of a text. Texts with similar meaning have nearby vectors, which is how semantic search finds them.
Long documents are split into passages of a few hundred tokens, so retrieval returns the relevant paragraph rather than a whole manual.
Keyword search (exact terms, product codes) and vector search (meaning) run together, and their results are merged.
A slower, more accurate model reorders the top candidates by reading the question and each passage together.
The answer points to the passages it used, so users can check them and you can measure whether each claim is supported.
Every chunk carries the access rules of its source document. Retrieval filters by the user's permissions before anything reaches the model.
Learn them properly in the deep dives:
Question
Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.
2.1Brief and scope
The question exactly as the interviewer asks it, and what they want to learn from your answer.
Our support and sales teams want an assistant that answers questions from 5 million internal documents: policies, product manuals, tickets and wiki pages. Answers must cite sources, respect who is allowed to see what, and stay current as documents change daily.
What it tests: The full RAG pipeline: ingestion, chunking, embeddings, hybrid retrieval, reranking, permissions, freshness and evaluation.
The answer in one sentence. Change-feed ingestion into a permission-aware hybrid index, retrieve-rerank-generate at query time, and citations users can check in one click.
How to spend the 45 minutes
- Clarify~5 minAsk questions, agree on scope and write down assumptions.
- Estimate~5 minRough numbers for users, data and traffic, and what they imply.
- High-level design~11 minComponents, data flow and the key API on the board.
- Deep dive~16 minThe two or three hardest parts, with real mechanisms.
- Trade-offs and wrap-up~9 minDecisions, failure modes, and what you would do next.
2.2Clarify the problem
The first five minutes. Each answer removes a guess from your design.
| You ask | Interviewer answers |
|---|---|
| Where do documents live? | Confluence, Google Drive, Zendesk and a policy portal. |
| Do permissions differ by document? | Yes, by team and region; some are restricted to managers. |
| How fresh must answers be? | Within an hour of a document changing. |
| Languages? | Mostly English, some Hindi. |
2.3Estimate the scale
Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.
| Metric | Estimate | How |
|---|---|---|
| Documents | 5M | |
| Chunks | ≈ 50M | ~10 chunks of 500 tokens each |
| Vector storage | ≈ 200 GB raw | 50M × 1,024 dims × 4 bytes; much less with int8 or PQ |
| Daily changes | ≈ 50k documents | re-embed only changed chunks |
| Queries | ≈ 20/s at peak | 5,000 users |
Detailed answer
The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.
3.1Architecture
Two pipelines: ingestion keeps a searchable, permission-aware index current; query time retrieves, filters, reranks and asks the model to answer only from what was retrieved.
- Connectors
- Use each source’s change feed or webhooks, not nightly full crawls, so changes arrive within minutes. Deleted documents must be deleted from the index too.
- Chunking
- Split along document structure (headings, sections, table boundaries) to roughly 300–800 tokens with a small overlap. Keep the title and section path with each chunk; it improves retrieval a lot.
- Hybrid index
- Dense vectors catch meaning; BM25 keyword search catches exact terms like error codes and product SKUs. Combine scores with reciprocal rank fusion.
- Permission filtering
- Each chunk stores the groups allowed to read it. The retriever filters by the user’s groups inside the search, before anything reaches the model.
- Reranker
- A cross-encoder rescores the top 50 candidates against the question and keeps the best 8. It is slower per item but far more accurate than vector similarity alone.
- Answer model
- Instructed to answer only from the provided chunks, cite them by ID, and say it does not know when they do not contain the answer.
3.2Key flow, step by step
“What is our refund policy for annual plans in the UAE?”
- The search itself applies the permission filter, so restricted text never enters the prompt.
- Citations map back to document URLs and section anchors so users can verify.
- If the top chunks score too low, the system answers ‘I couldn’t find this’ rather than guessing.
3.3Front end
The web front end turns retrieval into trust. Citations, source previews, honest ‘not found’ states and fast feedback matter as much as the model.
Screen map
- Answer
- Streams text and renders citation markers as interactive chips. The app drops unknown citation IDs and never shows broken links.
- Source preview
- A popover with the document title, section path, last updated date and the retrieved paragraph, with the supporting sentence highlighted.
- Not found
- Instead of guessing, says it couldn’t find this and lists the closest documents with links.
- Admin
- Index lag per source, recent failed queries and evaluation scores, so the platform team can see quality drift.
Client layers
| Layer | Technology | Responsibility |
|---|---|---|
| UI | React + TypeScript | Ask box, answer, citations, previews |
| Server state | TanStack Query | History, sources, feedback mutations |
| Streaming | fetch + ReadableStream (SSE) | Tokens and citation events |
| Markdown | Incremental renderer | Render text and replace [n] with chips |
| Auth | SSO session | Identity forwarded to the query API for ACL filtering |
Every state the user can see
| State | What the user sees | How it works |
|---|---|---|
| Searching | ‘Searching 3 sources…’ | Retrieval in progress |
| Answering | Text streams with citations | Generation in progress |
| Answered | Citations clickable; feedback buttons | Done event |
| Not found | Honest message + closest documents | Top scores below threshold |
| Restricted | ‘Some results are hidden by permissions’ | Retriever filtered matches |
| Stale source | Badge ‘Updated 2 hours ago’ | Source updated after indexing |
Code
Rendering citations as chips (React)
1export function Answer({ text, citations }: { text: string; citations: Record<string, Citation> }) {2 const parts = text.split(/(\[\d+\])/g); // keep [1], [2] as separate parts3 return (4 <p className="answer">5 {parts.map((part, i) => {6 const m = part.match(/^\[(\d+)\]$/);7 if (!m) return <span key={i}>{part}</span>;8 const c = citations[m[1]];9 if (!c) return null; // never show a citation we can't back10 return (11 <SourcePopover key={i} citation={c}>12 <button className="cite" aria-label={'Source ' + m[1] + ': ' + c.title}>{m[1]}</button>13 </SourcePopover>14 );15 })}16 </p>17 );18}Streaming the answer (TypeScript)
1const res = await fetch('/api/ask', { method: 'POST', body: JSON.stringify({ q }), signal: abort.signal });2const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();3let buf = '';4for (;;) {5 const { value, done } = await reader.read();6 if (done) break;7 buf += value;8 const events = buf.split('\n\n'); buf = events.pop()!; // SSE frames end with a blank line9 for (const e of events) dispatch(parseSse(e)); // token | citation | done10}3.4Data model and APIs
The tables and contracts that make the design concrete.
Chunk record
1{ "chunk_id": "doc-311#4", "doc_id": "doc-311", "title": "Refund policy",2 "section_path": ["Plans", "Annual", "Refunds"], "text": "…",3 "acl_groups": ["sales-mena", "support-l2"], "lang": "en",4 "updated_at": "2026-10-28T09:14:00Z", "embedding": [0.012, -0.044, …] }Hybrid query with ACL (SQL, pgvector)
1WITH v AS (2 SELECT chunk_id, row_number() OVER (ORDER BY embedding <=> $1) AS r3 FROM chunks WHERE acl_groups && $2 ORDER BY embedding <=> $1 LIMIT 50),4k AS (5 SELECT chunk_id, row_number() OVER (ORDER BY ts_rank(tsv, q) DESC) AS r6 FROM chunks, plainto_tsquery($3) q WHERE tsv @@ q AND acl_groups && $2 LIMIT 50)7SELECT chunk_id, sum(1.0 / (60 + r)) AS score -- reciprocal rank fusion8FROM (SELECT * FROM v UNION ALL SELECT * FROM k) s9GROUP BY chunk_id ORDER BY score DESC LIMIT 50;3.5Tested code, explained
The core of this design as working code, with what each part does and why.
Merging rankings with RRF
Keyword and vector scores aren't comparable. Ranks are.
1export interface RrfOptions {2 k?: number;3 weights?: number[];4 top?: number;5}6 7/** Reciprocal rank fusion: each list adds weight / (k + rank). Raw scores never enter the sum. */8export function rrf(lists: string[][], { k = 60, weights, top }: RrfOptions = {}): { id: string; score: number }[] {9 const scores = new Map<string, number>();10 lists.forEach((list, l) =>11 list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + (weights?.[l] ?? 1) / (k + i + 1)))12 );13 const ranked = [...scores].map(([id, score]) => ({ id, score })).sort((a, b) => b.score - a.score);14 return top === undefined ? ranked : ranked.slice(0, top);15}/ (k + i + 1)- Only position counts. An item near the top of both lists beats one at the top of just one.
k = 60- Softens the gap between top ranks. 60 is the value from the original paper.
Tested by npm run test:modules.
Retrieve widely, rerank precisely
A cross-encoder is accurate but slow, so it only sees the top candidates, and it's allowed to fail.
1export interface Candidate {2 id: string;3 text: string;4}5 6interface Options {7 candidates?: number;8 top?: number;9 timeoutMs?: number;10}11 12/** Retrieve widely, rerank precisely. If the reranker fails or runs late, keep the retrieval order. */13export async function retrieveAndRerank(14 query: string,15 retrieve: (q: string, n: number) => Promise<Candidate[]>,16 rerank: (q: string, docs: Candidate[]) => Promise<number[]>,17 { candidates = 100, top = 5, timeoutMs = 300 }: Options = {}18): Promise<{ results: Candidate[]; reranked: boolean }> {19 const docs = await retrieve(query, candidates);20 const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("rerank timeout")), timeoutMs));21 try {22 const scores = await Promise.race([rerank(query, docs), timeout]);23 const order = docs.map((d, i) => ({ d, s: scores[i] })).sort((a, b) => b.s - a.s);24 return { results: order.slice(0, top).map((x) => x.d), reranked: true };25 } catch {26 return { results: docs.slice(0, top), reranked: false };27 }28}candidates = 100- Cast the net wide first, because reranking can't recover what retrieval missed.
Promise.race([rerank(...), timeout])- Puts a limit on latency.
reranked: false- On failure, fall back to retrieval order and say so, rather than failing the request.
Tested by npm run test:modules.
Search small, read large
Small chunks match queries precisely, and large ones give the model enough context. This keeps both.
1export interface Chunk {2 id: string;3 parentId: string;4 text: string;5}6 7/** Splits a document into large parents, and each parent into small children to embed. */8export function splitDocument(docId: string, words: string[], parentSize: number, childSize: number) {9 const parents = new Map<string, string>();10 const children: Chunk[] = [];11 for (let p = 0; p * parentSize < words.length; p++) {12 const parentWords = words.slice(p * parentSize, (p + 1) * parentSize);13 const parentId = `${docId}#${p}`;14 parents.set(parentId, parentWords.join(" "));15 for (let c = 0; c * childSize < parentWords.length; c++) {16 children.push({ id: `${parentId}.${c}`, parentId, text: parentWords.slice(c * childSize, (c + 1) * childSize).join(" ") });17 }18 }19 return { parents, children };20}21 22/** Search hits children; the model reads parents. Each parent appears once, at its best score. */23export function toParents(hits: { child: Chunk; score: number }[], parents: Map<string, string>, limit: number) {24 const best = new Map<string, number>();25 for (const { child, score } of hits) best.set(child.parentId, Math.max(best.get(child.parentId) ?? -Infinity, score));26 return [...best]27 .sort((a, b) => b[1] - a[1])28 .slice(0, limit)29 .map(([id, score]) => ({ id, score, text: parents.get(id) ?? "" }));30}children- Only these get embedded and searched.
best.set(child.parentId, Math.max(...))- Several children of one parent collapse into one result, at its best score.
Tested by npm run test:modules.
Fitting context into a budget
Retrieval usually returns more than fits. This keeps the best sentences, in the order they were written.
1export interface Sentence {2 text: string;3 score: number;4}5 6/** Keeps the highest-scoring sentences that fit the budget, then restores reading order. */7export function fitContext(sentences: Sentence[], budget: number, count: (s: string) => number): string[] {8 const ranked = sentences.map((s, i) => ({ ...s, i })).sort((a, b) => b.score - a.score);9 const kept: { text: string; i: number }[] = [];10 let used = 0;11 for (const s of ranked) {12 const cost = count(s.text);13 if (used + cost > budget) continue;14 used += cost;15 kept.push(s);16 }17 return kept.sort((a, b) => a.i - b.i).map((s) => s.text);18}sort((a, b) => b.score - a.score)- Picks by relevance.
continue- Skips one that doesn't fit and tries smaller ones, instead of stopping.
sort((a, b) => a.i - b.i)- Restores reading order, because shuffled sentences confuse a model.
Tested by npm run test:modules.
3.6Hard parts, in depth
Where interviewers spend the second half of the round.
Measuring retrieval separately from answers
Build a test set of a few hundred questions with the documents that should answer them. Measure recall at k for retrieval on its own; most bad answers are retrieval failures, not model failures.
Then measure answers for faithfulness (only uses retrieved facts), relevance and citation accuracy, with an LLM judge checked against human ratings.
Freshness and deletes
Each chunk carries a content hash. On a change, re-chunk the document, re-embed only chunks whose hash changed, and delete the rest. Track index lag as a metric with an alert.
Scaling the index
At 50 million vectors, an HNSW index in a dedicated vector store or OpenSearch is typical; pgvector works well into the tens of millions with care. Quantising vectors to int8 cuts memory about four times with small recall loss. Shard by tenant or source if needed.
Q&A
The questions that come after the design, with model answers. Answer each one out loud before opening it.
4.1Follow-up questions
What the interviewer asks once the main design is on the board.
How do you handle tables and PDFs?
How do you choose an embedding model?
When is a long-context model enough without RAG?
4.2Front-end follow-ups
Questions aimed at the client: rendering, state and devices.
Why drop citations the model invents?
How does the UI help catch stale answers?
4.3Incidents, probes and traps
Realistic production incidents and pointed probes, grouped by topic.
The product 1
ProbeWhy not paste all documents into a long-context model?
Requirements 1
TrapWhich document question matters most?
Scale 1
ProbeFifty million vectors of 1,024 dimensions. How do you fit them?
Architecture 2
IncidentA sales rep got an answer quoting an HR document he cannot access.
ProbeWhy add a reranker?
The key flow 1
IncidentAnswers quote a policy that changed yesterday.
Front end 2
IncidentUsers trust answers whose citations don’t actually support the claim.
ProbeHow should the UI handle weak retrieval?
Data and APIs 1
TrapWhy reciprocal rank fusion instead of adding scores?
The hard parts 2
IncidentRecall collapsed after switching embedding models.
ProbeHow do you handle tables?
Trade-offs 1
ProbeWhen would you fine-tune instead of using RAG?
Failures 2
IncidentA wiki page tells the model to reveal salaries.
IncidentIngestion fell two days behind after a source API changed its rate limits.
Extra depth
What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.
5.1Trade-offs and decisions
Every choice has a cost. Naming it is what interviewers listen for.
Show detailsHide details
| Decision | Chosen | Rejected | Why |
|---|---|---|---|
| Retrieval | Hybrid vector + BM25 | Vectors only | Exact identifiers and codes are common in support content. |
| Permissions | Filter at search time | Filter after generation | Restricted text must never reach the model. |
| Precision | Reranker on top 50 | Larger top-k into the prompt | Better answers with fewer tokens. |
| Fine-tuning vs RAG | RAG | Fine-tune on documents | Documents change daily and need citations and permissions. |
5.2Failure modes
What breaks in production, and how the design keeps working when it does.
Show detailsHide details
| What breaks | How the design copes |
|---|---|
| Stale or deleted content returned | Change-feed ingestion, delete propagation, index-lag alerts. |
| Confident wrong answer | Low-score threshold returns ‘not found’; faithfulness checks in evaluation. |
| Permission change not applied | ACL sync on group changes, and a periodic full reconcile. |
| Prompt injection inside a document | Treat chunk text as data, strip instructions, and never give the answer model tools that write. |
5.3Performance and accessibility
The client-side details that separate a demo from a product.
Performance
- Show the ‘searching’ state immediately; retrieval often takes longer than the first token.
- Cache source previews by chunk ID.
- Debounce suggestions while typing; never call the model on keystrokes.
Accessibility and localisation
- Citation chips are buttons with descriptive labels.
- Popovers are keyboard reachable and dismissable with Escape.
- Answers keep a readable line length and proper headings.
5.4Senior versus principal
The same design, answered at two levels. Use the rubric to score yourself.
A senior answer embeds documents into a vector database and passes the top results to a model.
A principal answer designs ingestion with change feeds and deletes, structure-aware chunking, hybrid retrieval with reranking, permission filtering inside search, and an evaluation plan that separates retrieval from generation.
Score yourself
5.5Whiteboard
Draw the design here, in this order. It stays in this browser.
- Ingestion pipeline left to right
- Hybrid index with ACL tags
- Query path: rewrite → retrieve → rerank → generate
- Citations flowing back to the user
- Evaluation and feedback loop