/Learn
Modules · Evals and guardrails · Coding pattern

Faithfulness, Answer Relevance & Context Recall Metrics

PracticeCoding patternTier 1 · Whole pictureAnalyze12 minEvals and guardrails

You will be able to: compute the four RAGAS metrics, and use them to tell a retrieval bug from a generation bug.

Source: Es et al., 2023 (RAGAS)

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A bad RAG answer has two possible causes: retrieval missed the evidence, or the model ignored it. Four metrics separate them. Context metrics judge retrieval; faithfulness and relevance judge generation.

Why it matters

Separates retrieval failures from generation failures, so you fix the right half of a RAG system.

1.2Key concepts

One idea per card, in plain words.

01Faithfulness

Share of the answer's claims that the retrieved context supports.

02Answer relevance

Does the answer address the question? Compares questions regenerated from the answer with the real one.

03Context recall

Share of the reference answer's claims that the context can support.

04Context precision

Do relevant chunks rank near the top?

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Users report wrong answers from your RAG bot. Compute metrics that tell you whether to fix retrieval or generation.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Build a test set

    Collect real questions with reference answers, including ones with no answer.

  2. Split into claims

    An LLM splits the answer into atomic claims, then checks each against the context.

  3. Score retrieval

    Compute context recall and precision from reference claims and chunk relevance.

  4. Score generation

    Compute faithfulness and answer relevance.

  5. Read the pattern

    Use the table below to find the failing half.

3.2Read the pattern

Context recallFaithfulnessDiagnosisFix
LowHighEvidence never arrivedChunking, hybrid search
HighLowModel ignored the evidencePrompt, citations, model
LowLowBoth brokenFix retrieval first
HighHigh, users unhappyWrong question answeredCheck answer relevance

3.3Code

Working code for the core of it.

Scoring retrieval and answers

Together these separate a retrieval failure from a generation failure.

ragas-metrics.tsTypeScript
1const mean = (xs: number[]) => (xs.length ? xs.reduce((a, b) => a + b, 0) / xs.length : 0);2const cosine = (a: number[], b: number[]) => a.reduce((s, x, i) => s + x * b[i], 0) / (Math.hypot(...a) * Math.hypot(...b));3 4/** Faithfulness: share of the answer's claims that the retrieved context supports. */5export const faithfulness = (supported: boolean[]) => mean(supported.map(Number));6 7/** Context recall: share of the reference answer's claims that the context can attribute. */8export const contextRecall = (attributed: boolean[]) => mean(attributed.map(Number));9 10/** Context precision: mean precision@k over the ranks that hold a relevant chunk. */11export function contextPrecision(relevantAtRank: boolean[]): number {12  let hits = 0;13  const precisions: number[] = [];14  relevantAtRank.forEach((relevant, i) => {15    if (relevant) precisions.push(++hits / (i + 1));16  });17  return mean(precisions);18}19 20/** Answer relevance: questions regenerated from the answer should resemble the real question. */21export const answerRelevance = (question: number[], generated: number[][]) => mean(generated.map((g) => cosine(question, g)));
21 linesUTF-8TypeScript
faithfulness
Did the answer stick to the context?
contextRecall
Did retrieval find what was needed?
contextPrecision
Were the useful chunks ranked high?
answerRelevance
Does the answer address the question?

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Do these need labelled data?
Faithfulness and answer relevance don't. Context recall needs reference answers.
Why not one overall score?
An average hides which half failed, and that is the whole point.
How noisy are they?
They use LLM judgements, so sample sizes matter. Look at trends and failing examples, not small differences.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Unanswerable questions
Include them. A good system says it doesn't know.
Counterpoint
High faithfulness can hide a useless answer that merely repeats the context. Pair metrics with human review.
AI Engineer · Faithfulness, Answer Relevance & Context Recall Metrics