/Interview
Interview room · AI system design · 45 minutes

Retrieval-augmented answers over company documents

PracticeSystem designTier 3 · Edge casesCreate45 minRAG and retrieval

You will be able to: design retrieval-augmented answers over company documents in 45 minutes, and defend each trade-off.

AI engineeringDesign rounds

Study with AI

Ready45:00
ClarifyEstimateHigh-level designDeep diveTrade-offs and wrap-up
Part 1 of 5

Basics

Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.

1.1Product in plain words

Who uses it, what a normal day looks like, and why it is harder than it seems.

An internal assistant for support and sales staff that answers questions using the company’s own documents (policies, manuals, tickets and wiki pages) and always shows where each answer came from.

Who uses it

  • Support agents: need correct answers fast while a customer waits.
  • Sales staff: need the latest pricing and policy, by region.
  • Compliance: needs restricted documents to stay restricted.

A day in the life

  1. An agent asks: ‘What’s the refund policy for annual plans in the UAE?’
  2. The system searches 5 million documents, but only those she is allowed to see.
  3. The answer appears with numbered citations like [1] and [2].
  4. Hovering a citation shows the exact paragraph from the policy, with a link.
  5. She marks it helpful; a wrong answer would be flagged and added to the test set.

Why it is hard

Finding the right paragraph

Among millions of documents, retrieval quality decides answer quality.

Permissions

An answer must never be built from a document the user can’t open.

Freshness

Policies change; yesterday’s answer can be wrong today.

Trust

Users need to verify answers quickly, or they stop using it.

Think of it like this

Like a librarian who first fetches the three most relevant pages from the shelves you’re allowed into, then answers using only those pages, with sticky notes showing where each fact came from.

1.2Key concepts

The building blocks this design uses. Each one appears again in the answer.

01Embeddings

A vector that represents the meaning of a text. Texts with similar meaning have nearby vectors, which is how semantic search finds them.

02Chunking

Long documents are split into passages of a few hundred tokens, so retrieval returns the relevant paragraph rather than a whole manual.

03Hybrid search

Keyword search (exact terms, product codes) and vector search (meaning) run together, and their results are merged.

04Reranking

A slower, more accurate model reorders the top candidates by reading the question and each passage together.

05Citations

The answer points to the passages it used, so users can check them and you can measure whether each claim is supported.

06Permission filtering

Every chunk carries the access rules of its source document. Retrieval filters by the user's permissions before anything reaches the model.

Learn them properly in the deep dives:

Part 2 of 5

Question

Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.

2.1Brief and scope

The question exactly as the interviewer asks it, and what they want to learn from your answer.

Interviewer

Our support and sales teams want an assistant that answers questions from 5 million internal documents: policies, product manuals, tickets and wiki pages. Answers must cite sources, respect who is allowed to see what, and stay current as documents change daily.

What it tests: The full RAG pipeline: ingestion, chunking, embeddings, hybrid retrieval, reranking, permissions, freshness and evaluation.

The answer in one sentence. Change-feed ingestion into a permission-aware hybrid index, retrieve-rerank-generate at query time, and citations users can check in one click.

How to spend the 45 minutes

  1. Clarify~5 minAsk questions, agree on scope and write down assumptions.
  2. Estimate~5 minRough numbers for users, data and traffic, and what they imply.
  3. High-level design~11 minComponents, data flow and the key API on the board.
  4. Deep dive~16 minThe two or three hardest parts, with real mechanisms.
  5. Trade-offs and wrap-up~9 minDecisions, failure modes, and what you would do next.

2.2Clarify the problem

The first five minutes. Each answer removes a guess from your design.

You askInterviewer answers
Where do documents live?Confluence, Google Drive, Zendesk and a policy portal.
Do permissions differ by document?Yes, by team and region; some are restricted to managers.
How fresh must answers be?Within an hour of a document changing.
Languages?Mostly English, some Hindi.

2.3Estimate the scale

Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.

MetricEstimateHow
Documents5M
Chunks≈ 50M~10 chunks of 500 tokens each
Vector storage≈ 200 GB raw50M × 1,024 dims × 4 bytes; much less with int8 or PQ
Daily changes≈ 50k documentsre-embed only changed chunks
Queries≈ 20/s at peak5,000 users
Part 3 of 5

Detailed answer

The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.

3.1Architecture

Two pipelines: ingestion keeps a searchable, permission-aware index current; query time retrieves, filters, reranks and asks the model to answer only from what was retrieved.

INGESTIONQUERY TIMEConnectorsDrive · Confluence· ZendeskACL syncgroups → doc idsParse + cleantext · tables · OCRHybrid indexvectors + BM25 +ACLChunkerby headings, 500tokEmbedderbatch GPU jobQuery APIuser + questionQuery rewriteexpand · translateRetrieverfilter by ACL, top50Rerankercross-encoder, top8Eval + feedbacktraces · scoresAnswer modelcite or say unknownchangesupsertpermissionssearchtop 8trace
ServiceData storeAI componentExternal
Reference design. Boxes show components; arrows show who calls whom.
Connectors
Use each source’s change feed or webhooks, not nightly full crawls, so changes arrive within minutes. Deleted documents must be deleted from the index too.
Chunking
Split along document structure (headings, sections, table boundaries) to roughly 300–800 tokens with a small overlap. Keep the title and section path with each chunk; it improves retrieval a lot.
Hybrid index
Dense vectors catch meaning; BM25 keyword search catches exact terms like error codes and product SKUs. Combine scores with reciprocal rank fusion.
Permission filtering
Each chunk stores the groups allowed to read it. The retriever filters by the user’s groups inside the search, before anything reaches the model.
Reranker
A cross-encoder rescores the top 50 candidates against the question and keeps the best 8. It is slower per item but far more accurate than vector similarity alone.
Answer model
Instructed to answer only from the provided chunks, cite them by ID, and say it does not know when they do not contain the answer.

3.2Key flow, step by step

“What is our refund policy for annual plans in the UAE?”

UserQuery APIRetrieverRerankerAnswer modelquestion + identityrewrite: refund annual plan UAEhybrid search, groups = sales-mena50 chunksscore 50 vs questiontop 8answer from these, cite idsanswer + [doc-311 §4]answer with links
Solid arrows are requests; dashed arrows are responses.
  1. The search itself applies the permission filter, so restricted text never enters the prompt.
  2. Citations map back to document URLs and section anchors so users can verify.
  3. If the top chunks score too low, the system answers ‘I couldn’t find this’ rather than guessing.

3.3Front end

The web front end turns retrieval into trust. Citations, source previews, honest ‘not found’ states and fast feedback matter as much as the model.

Screen map

Askquestion box ·filtersAnswerstreaming ·citationsSource previewexact paragraphOpen documentin source appHistorysaved answersNot foundsuggest sourcesFeedbackwrong · outdatedAdminsources · lag ·evalhover [1]openlow score
ClientExternal
Screens and how users move between them.
Answer
Streams text and renders citation markers as interactive chips. The app drops unknown citation IDs and never shows broken links.
Source preview
A popover with the document title, section path, last updated date and the retrieved paragraph, with the supporting sentence highlighted.
Not found
Instead of guessing, says it couldn’t find this and lists the closest documents with links.
Admin
Index lag per source, recent failed queries and evaluation scores, so the platform team can see quality drift.

Client layers

LayerTechnologyResponsibility
UIReact + TypeScriptAsk box, answer, citations, previews
Server stateTanStack QueryHistory, sources, feedback mutations
Streamingfetch + ReadableStream (SSE)Tokens and citation events
MarkdownIncremental rendererRender text and replace [n] with chips
AuthSSO sessionIdentity forwarded to the query API for ACL filtering

Every state the user can see

StateWhat the user seesHow it works
Searching‘Searching 3 sources…’Retrieval in progress
AnsweringText streams with citationsGeneration in progress
AnsweredCitations clickable; feedback buttonsDone event
Not foundHonest message + closest documentsTop scores below threshold
Restricted‘Some results are hidden by permissions’Retriever filtered matches
Stale sourceBadge ‘Updated 2 hours ago’Source updated after indexing

Code

Rendering citations as chips (React)

rendering-citations-as.tsxTypeScript JSX
1export function Answer({ text, citations }: { text: string; citations: Record<string, Citation> }) {2  const parts = text.split(/(\[\d+\])/g);                  // keep [1], [2] as separate parts3  return (4    <p className="answer">5      {parts.map((part, i) => {6        const m = part.match(/^\[(\d+)\]$/);7        if (!m) return <span key={i}>{part}</span>;8        const c = citations[m[1]];9        if (!c) return null;                                   // never show a citation we can't back10        return (11          <SourcePopover key={i} citation={c}>12            <button className="cite" aria-label={'Source ' + m[1] + ': ' + c.title}>{m[1]}</button>13          </SourcePopover>14        );15      })}16    </p>17  );18}
18 linesUTF-8TypeScript JSX

Streaming the answer (TypeScript)

streaming-answer-typescript.tsTypeScript
1const res = await fetch('/api/ask', { method: 'POST', body: JSON.stringify({ q }), signal: abort.signal });2const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();3let buf = '';4for (;;) {5  const { value, done } = await reader.read();6  if (done) break;7  buf += value;8  const events = buf.split('\n\n'); buf = events.pop()!;   // SSE frames end with a blank line9  for (const e of events) dispatch(parseSse(e));            // token | citation | done10}
10 linesUTF-8TypeScript

3.4Data model and APIs

The tables and contracts that make the design concrete.

Chunk record

chunk-record.jsonJSON
1{ "chunk_id": "doc-311#4", "doc_id": "doc-311", "title": "Refund policy",2  "section_path": ["Plans", "Annual", "Refunds"], "text": "…",3  "acl_groups": ["sales-mena", "support-l2"], "lang": "en",4  "updated_at": "2026-10-28T09:14:00Z", "embedding": [0.012, -0.044, …] }
4 linesUTF-8JSON

Hybrid query with ACL (SQL, pgvector)

hybrid-query-acl.sqlSQL
1WITH v AS (2  SELECT chunk_id, row_number() OVER (ORDER BY embedding <=> $1) AS r3  FROM chunks WHERE acl_groups && $2 ORDER BY embedding <=> $1 LIMIT 50),4k AS (5  SELECT chunk_id, row_number() OVER (ORDER BY ts_rank(tsv, q) DESC) AS r6  FROM chunks, plainto_tsquery($3) q WHERE tsv @@ q AND acl_groups && $2 LIMIT 50)7SELECT chunk_id, sum(1.0 / (60 + r)) AS score          -- reciprocal rank fusion8FROM (SELECT * FROM v UNION ALL SELECT * FROM k) s9GROUP BY chunk_id ORDER BY score DESC LIMIT 50;
9 linesUTF-8SQL

3.5Tested code, explained

The core of this design as working code, with what each part does and why.

Merging rankings with RRF

Keyword and vector scores aren't comparable. Ranks are.

rrf.tsTypeScript
1export interface RrfOptions {2  k?: number;3  weights?: number[];4  top?: number;5}6 7/** Reciprocal rank fusion: each list adds weight / (k + rank). Raw scores never enter the sum. */8export function rrf(lists: string[][], { k = 60, weights, top }: RrfOptions = {}): { id: string; score: number }[] {9  const scores = new Map<string, number>();10  lists.forEach((list, l) =>11    list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + (weights?.[l] ?? 1) / (k + i + 1)))12  );13  const ranked = [...scores].map(([id, score]) => ({ id, score })).sort((a, b) => b.score - a.score);14  return top === undefined ? ranked : ranked.slice(0, top);15}
15 linesUTF-8TypeScript
/ (k + i + 1)
Only position counts. An item near the top of both lists beats one at the top of just one.
k = 60
Softens the gap between top ranks. 60 is the value from the original paper.

Tested by npm run test:modules.

Retrieve widely, rerank precisely

A cross-encoder is accurate but slow, so it only sees the top candidates, and it's allowed to fail.

rerank-pipeline.tsTypeScript
1export interface Candidate {2  id: string;3  text: string;4}5 6interface Options {7  candidates?: number;8  top?: number;9  timeoutMs?: number;10}11 12/** Retrieve widely, rerank precisely. If the reranker fails or runs late, keep the retrieval order. */13export async function retrieveAndRerank(14  query: string,15  retrieve: (q: string, n: number) => Promise<Candidate[]>,16  rerank: (q: string, docs: Candidate[]) => Promise<number[]>,17  { candidates = 100, top = 5, timeoutMs = 300 }: Options = {}18): Promise<{ results: Candidate[]; reranked: boolean }> {19  const docs = await retrieve(query, candidates);20  const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("rerank timeout")), timeoutMs));21  try {22    const scores = await Promise.race([rerank(query, docs), timeout]);23    const order = docs.map((d, i) => ({ d, s: scores[i] })).sort((a, b) => b.s - a.s);24    return { results: order.slice(0, top).map((x) => x.d), reranked: true };25  } catch {26    return { results: docs.slice(0, top), reranked: false };27  }28}
28 linesUTF-8TypeScript
candidates = 100
Cast the net wide first, because reranking can't recover what retrieval missed.
Promise.race([rerank(...), timeout])
Puts a limit on latency.
reranked: false
On failure, fall back to retrieval order and say so, rather than failing the request.

Tested by npm run test:modules.

Search small, read large

Small chunks match queries precisely, and large ones give the model enough context. This keeps both.

parent-child.tsTypeScript
1export interface Chunk {2  id: string;3  parentId: string;4  text: string;5}6 7/** Splits a document into large parents, and each parent into small children to embed. */8export function splitDocument(docId: string, words: string[], parentSize: number, childSize: number) {9  const parents = new Map<string, string>();10  const children: Chunk[] = [];11  for (let p = 0; p * parentSize < words.length; p++) {12    const parentWords = words.slice(p * parentSize, (p + 1) * parentSize);13    const parentId = `${docId}#${p}`;14    parents.set(parentId, parentWords.join(" "));15    for (let c = 0; c * childSize < parentWords.length; c++) {16      children.push({ id: `${parentId}.${c}`, parentId, text: parentWords.slice(c * childSize, (c + 1) * childSize).join(" ") });17    }18  }19  return { parents, children };20}21 22/** Search hits children; the model reads parents. Each parent appears once, at its best score. */23export function toParents(hits: { child: Chunk; score: number }[], parents: Map<string, string>, limit: number) {24  const best = new Map<string, number>();25  for (const { child, score } of hits) best.set(child.parentId, Math.max(best.get(child.parentId) ?? -Infinity, score));26  return [...best]27    .sort((a, b) => b[1] - a[1])28    .slice(0, limit)29    .map(([id, score]) => ({ id, score, text: parents.get(id) ?? "" }));30}
30 linesUTF-8TypeScript
children
Only these get embedded and searched.
best.set(child.parentId, Math.max(...))
Several children of one parent collapse into one result, at its best score.

Tested by npm run test:modules.

Fitting context into a budget

Retrieval usually returns more than fits. This keeps the best sentences, in the order they were written.

context-budget.tsTypeScript
1export interface Sentence {2  text: string;3  score: number;4}5 6/** Keeps the highest-scoring sentences that fit the budget, then restores reading order. */7export function fitContext(sentences: Sentence[], budget: number, count: (s: string) => number): string[] {8  const ranked = sentences.map((s, i) => ({ ...s, i })).sort((a, b) => b.score - a.score);9  const kept: { text: string; i: number }[] = [];10  let used = 0;11  for (const s of ranked) {12    const cost = count(s.text);13    if (used + cost > budget) continue;14    used += cost;15    kept.push(s);16  }17  return kept.sort((a, b) => a.i - b.i).map((s) => s.text);18}
18 linesUTF-8TypeScript
sort((a, b) => b.score - a.score)
Picks by relevance.
continue
Skips one that doesn't fit and tries smaller ones, instead of stopping.
sort((a, b) => a.i - b.i)
Restores reading order, because shuffled sentences confuse a model.

Tested by npm run test:modules.

3.6Hard parts, in depth

Where interviewers spend the second half of the round.

Measuring retrieval separately from answers

Build a test set of a few hundred questions with the documents that should answer them. Measure recall at k for retrieval on its own; most bad answers are retrieval failures, not model failures.

Then measure answers for faithfulness (only uses retrieved facts), relevance and citation accuracy, with an LLM judge checked against human ratings.

Freshness and deletes

Each chunk carries a content hash. On a change, re-chunk the document, re-embed only chunks whose hash changed, and delete the rest. Track index lag as a metric with an alert.

Scaling the index

At 50 million vectors, an HNSW index in a dedicated vector store or OpenSearch is typical; pgvector works well into the tens of millions with care. Quantising vectors to int8 cuts memory about four times with small recall loss. Shard by tenant or source if needed.

Part 4 of 5

Q&A

The questions that come after the design, with model answers. Answer each one out loud before opening it.

4.1Follow-up questions

What the interviewer asks once the main design is on the board.

How do you handle tables and PDFs?
Layout-aware parsing; tables become row-wise text with headers repeated, and OCR for scans with a quality flag.
How do you choose an embedding model?
Benchmark candidates on your own test set for recall, cost and multilingual quality, not on public leaderboards alone.
When is a long-context model enough without RAG?
For a few documents per question. At millions of documents, with permissions and freshness, retrieval is still required.

4.2Front-end follow-ups

Questions aimed at the client: rendering, state and devices.

Why drop citations the model invents?
Every citation must map to a chunk the retriever actually returned; anything else would mislead users into trusting an unsupported claim.
How does the UI help catch stale answers?
It shows each source’s last updated time and flags sources changed after indexing, and one-click feedback reports outdated answers.

4.3Incidents, probes and traps

Realistic production incidents and pointed probes, grouped by topic.

The product 1

ProbeWhy not paste all documents into a long-context model?
Five million documents don’t fit, cost would explode, permissions couldn’t be enforced, and freshness would require re-sending everything.

Requirements 1

TrapWhich document question matters most?
How permissions work and whether sources offer change feeds. They decide whether the system is safe and fresh.

Scale 1

ProbeFifty million vectors of 1,024 dimensions. How do you fit them?
About 200 GB raw. Quantise to int8 or product quantisation, shard by tenant or source, and size HNSW memory overhead, or use a managed vector store.

Architecture 2

IncidentA sales rep got an answer quoting an HR document he cannot access.
The system filtered permissions after retrieval, or the ACL sync lagged. Filter inside the search, sync permission changes immediately, run periodic full reconciliations, and audit every answer’s sources.
ProbeWhy add a reranker?
Vector similarity is approximate. A cross-encoder reads the question and each candidate together and is far more precise for the final top few.

The key flow 1

IncidentAnswers quote a policy that changed yesterday.
The index was stale or old chunks were not deleted. Use change feeds, delete replaced chunks, version documents, and alert on index lag.

Front end 2

IncidentUsers trust answers whose citations don’t actually support the claim.
Highlight the supporting sentence in the source preview, run faithfulness checks, and drop citations that don’t map to retrieved chunks.
ProbeHow should the UI handle weak retrieval?
Say it couldn’t find the answer and show the closest documents, rather than letting the model guess.

Data and APIs 1

TrapWhy reciprocal rank fusion instead of adding scores?
Keyword and vector scores aren’t on comparable scales. Fusing by rank is robust without tuning.

The hard parts 2

IncidentRecall collapsed after switching embedding models.
One index mixed old and new vectors. Re-embed everything into a new index, dual-run, compare on the test set and switch over.
ProbeHow do you handle tables?
Parse layout, convert rows to text with headers repeated in each row, and keep table title and section path with every chunk.

Trade-offs 1

ProbeWhen would you fine-tune instead of using RAG?
When the problem is behaviour or format rather than missing facts. Changing facts, citations and permissions all favour RAG.

Failures 2

IncidentA wiki page tells the model to reveal salaries.
Prompt injection in content. Treat chunks as data, give the answer model no write tools, rely on ACLs so salary data is never retrieved for unauthorised users, and filter outputs.
IncidentIngestion fell two days behind after a source API changed its rate limits.
Back off and resume from change-feed checkpoints, alert on index lag per source, and show source freshness in the UI.
Part 5 of 5

Extra depth

What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.

5.1Trade-offs and decisions

Every choice has a cost. Naming it is what interviewers listen for.

Show detailsHide details
DecisionChosenRejectedWhy
RetrievalHybrid vector + BM25Vectors onlyExact identifiers and codes are common in support content.
PermissionsFilter at search timeFilter after generationRestricted text must never reach the model.
PrecisionReranker on top 50Larger top-k into the promptBetter answers with fewer tokens.
Fine-tuning vs RAGRAGFine-tune on documentsDocuments change daily and need citations and permissions.

5.2Failure modes

What breaks in production, and how the design keeps working when it does.

Show detailsHide details
What breaksHow the design copes
Stale or deleted content returnedChange-feed ingestion, delete propagation, index-lag alerts.
Confident wrong answerLow-score threshold returns ‘not found’; faithfulness checks in evaluation.
Permission change not appliedACL sync on group changes, and a periodic full reconcile.
Prompt injection inside a documentTreat chunk text as data, strip instructions, and never give the answer model tools that write.

5.3Performance and accessibility

The client-side details that separate a demo from a product.

Performance

  • Show the ‘searching’ state immediately; retrieval often takes longer than the first token.
  • Cache source previews by chunk ID.
  • Debounce suggestions while typing; never call the model on keystrokes.

Accessibility and localisation

  • Citation chips are buttons with descriptive labels.
  • Popovers are keyboard reachable and dismissable with Escape.
  • Answers keep a readable line length and proper headings.

5.4Senior versus principal

The same design, answered at two levels. Use the rubric to score yourself.

Senior answer

A senior answer embeds documents into a vector database and passes the top results to a model.

Principal answer

A principal answer designs ingestion with change feeds and deletes, structure-aware chunking, hybrid retrieval with reranking, permission filtering inside search, and an evaluation plan that separates retrieval from generation.

Score yourself

5.5Whiteboard

Draw the design here, in this order. It stays in this browser.

  1. Ingestion pipeline left to right
  2. Hybrid index with ACL tags
  3. Query path: rewrite → retrieve → rerank → generate
  4. Citations flowing back to the user
  5. Evaluation and feedback loop
Tip: draw boxes first, then arrows, then label them with Text.
AI Engineer · interview room