Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
A bad RAG answer has two possible causes: retrieval missed the evidence, or the model ignored it. Four metrics separate them. Context metrics judge retrieval; faithfulness and relevance judge generation.
Separates retrieval failures from generation failures, so you fix the right half of a RAG system.
1.2Key concepts
One idea per card, in plain words.
Share of the answer's claims that the retrieved context supports.
Does the answer address the question? Compares questions regenerated from the answer with the real one.
Share of the reference answer's claims that the context can support.
Do relevant chunks rank near the top?
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Users report wrong answers from your RAG bot. Compute metrics that tell you whether to fix retrieval or generation.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Build a test set
Collect real questions with reference answers, including ones with no answer.
- Split into claims
An LLM splits the answer into atomic claims, then checks each against the context.
- Score retrieval
Compute context recall and precision from reference claims and chunk relevance.
- Score generation
Compute faithfulness and answer relevance.
- Read the pattern
Use the table below to find the failing half.
3.2Read the pattern
| Context recall | Faithfulness | Diagnosis | Fix |
|---|---|---|---|
| Low | High | Evidence never arrived | Chunking, hybrid search |
| High | Low | Model ignored the evidence | Prompt, citations, model |
| Low | Low | Both broken | Fix retrieval first |
| High | High, users unhappy | Wrong question answered | Check answer relevance |
3.3Code
Working code for the core of it.
Scoring retrieval and answers
Together these separate a retrieval failure from a generation failure.
1const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0);2const cosine = (a: number[], b: number[]) => a.reduce((s, x, i) => s + x * b[i], 0) / (Math.hypot(...a) * Math.hypot(...b));3 4/** Faithfulness: share of the answer's claims that the retrieved context supports. */5export const faithfulness = (supported: boolean[]) => mean(supported.map(Number));6 7/** Context recall: share of the reference answer's claims that the context can attribute. */8export const contextRecall = (attributed: boolean[]) => mean(attributed.map(Number));9 10/** Context precision: mean precision@k over the ranks that hold a relevant chunk. */11export function contextPrecision(relevantAtRank: boolean[]): number {12 let hits = 0;13 const precisions: number[] = [];14 relevantAtRank.forEach((relevant, i) => {15 if (relevant) precisions.push(++hits / (i + 1));16 });17 return mean(precisions);18}19 20/** Answer relevance: questions regenerated from the answer should resemble the real question. */21export const answerRelevance = (question: number[], generated: number[][]) => mean(generated.map((g) => cosine(question, g)));faithfulness- Did the answer stick to the context?
contextRecall- Did retrieval find what was needed?
contextPrecision- Were the useful chunks ranked high?
answerRelevance- Does the answer address the question?
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Do these need labelled data?
Why not one overall score?
How noisy are they?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Unanswerable questions
- Include them. A good system says it doesn't know.
- Counterpoint
- High faithfulness can hide a useless answer that merely repeats the context. Pair metrics with human review.