/Learn
Modules · Evals and guardrails · Long read

LLM-as-a-Judge Evaluation & Calibration Architecture

LearnLong readTier 1 · Whole pictureEvaluate25 minEvals and guardrails

You will be able to: design a judge-based eval, remove its known biases, and calibrate it against human labels.

Prove it: Faithfulness, Answer Relevance & Context Recall Metrics

Source: Zheng et al., 2023 (MT-Bench and Chatbot Arena)

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

People can't review every prompt change, so an LLM scores answers instead. It works well at scale, with known flaws: judges favour the first answer, longer answers and answers like their own. Good design removes those biases and checks the judge against people.

Why it matters

LLM judges scale evaluation, but only once you correct position, verbosity and self-enhancement bias.

1.2Key concepts

One idea per card, in plain words.

01Pairwise judging

Asks which of two answers wins. Fewer scale problems than absolute scores.

02Position bias

Judges favour one position. Ask twice with the answers swapped.

03Verbosity bias

Judges favour longer answers even when they say no more.

04Self-enhancement

A judge favours its own model's answers. Use a different model to judge.

05Cohen's kappa

Measures agreement beyond chance, between a judge and human labels.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Three teams ship prompt changes weekly and argue about quality. Design an automated judge they can trust as a release signal.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Who writes the rubric?Each team, for its own feature, reviewed against real failures.
What decides a release?Judge scores gate on a regression; people review borderline changes.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Write a rubric

    Name the criteria, with examples of good and bad answers. Vague rubrics give noisy scores.

  2. Ask for reasoning first

    Have the judge reason before its verdict, then parse a fixed output format.

  3. Swap positions

    Run each pair both ways. Count a win only when both runs agree.

  4. Calibrate with people

    Label a few hundred pairs by hand, and compute agreement with the judge.

  5. Watch for drift

    Re-run the calibration set whenever the judge model or rubric changes.

3.2Code

Working code for the core of it.

Checking an LLM judge

A judge you haven't measured is an opinion. These two functions make it a measurement.

judge-calibration.tsTypeScript
1/** Cohen's kappa: agreement between two raters, beyond what chance would produce. */2export function cohenKappa(a: string[], b: string[]): number {3  const n = a.length;4  const labels = new Set([...a, ...b]);5  const observed = a.filter((x, i) => x === b[i]).length / n;6  let chance = 0;7  for (const l of labels) chance += (a.filter((x) => x === l).length / n) * (b.filter((x) => x === l).length / n);8  return chance === 1 ? 1 : (observed - chance) / (1 - chance);9}10 11export type Verdict = "first" | "second" | "tie";12 13/** Removes position bias: ask twice with the answers swapped; a verdict counts only if both agree. */14export async function debiasedVerdict(judge: (x: string, y: string) => Promise<Verdict>, a: string, b: string): Promise<"a" | "b" | "tie"> {15  const [one, two] = await Promise.all([judge(a, b), judge(b, a)]);16  if (one === "first" && two === "second") return "a";17  if (one === "second" && two === "first") return "b";18  return "tie";19}
19 linesUTF-8TypeScript
cohenKappa
Agreement with human labels, beyond what chance would give.
Promise.all([judge(a, b), judge(b, a)])
Judges favour one position, so ask both ways and count only agreement.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

How good can a judge get?
In the MT-Bench study, GPT-4 agreed with humans 85% of the time, above the 81% humans agreed with each other. Your task may differ, so calibrate.
How do you handle verbosity bias?
Add a concision criterion to the rubric, and test the judge with padded copies of the same answer.
Which checks should run before a judge?
Exact checks first: schema, citations, banned phrases. They cost nothing and never drift.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Judge cost
Two judge calls per pair add up. Sample in production, and run in full on the regression set.
Counterpoint
A judge measures what its rubric names and misses what it doesn't. Keep reading real conversations.
AI Engineer · LLM-as-a-Judge Evaluation & Calibration Architecture