/Interview

AI interview rounds · Lesson 7 of 8 · 25 min

The evals round

Study with AI

Part 1Basics

Normal code either passes a test or it doesn't. A model's output varies, and 'good' is often a judgment call. Evals are how you get a number anyway: a set of examples, a way to grade them, and a score you can compare from one version to the next.

Tian Pan argues that the question to ask candidates is how they'd design an eval before they've touched a prompt, since it shows whether they understand the failure modes. Whether or not you face that exact round, it's a strong way to prepare.

Think of evals as tests plus statistics. Two things catch people out. A small set gives a wide range of uncertainty. And an average can improve while one group of users gets much worse.

Words you need

Golden set
Examples with known good outcomes, kept under version control.
Slice
A group of cases, such as billing questions or non-English users. Check each one.
LLM judge
A model used to grade outputs. Check it against human labels before you trust it.
Cohen's kappa
Agreement after removing what chance would produce. 0 is luck, and 1 is perfect.
Regression
Something that worked before and doesn't now.

A worked example

The prompt: 'Our support bot passes 95% of evals, but users are complaining. What do you do?' Meera doesn't touch the bot. First she asks where the eval cases came from. They were written by the team a year ago, before most of the current customers arrived, and hold few of today's billing questions. She samples real traffic, labels 200 cases, and finds billing at 60%. The bot didn't get worse. The eval had stopped matching reality.

Part 2The question

How do you prove an AI feature works, and that the next change didn't break it?

By the end of this lesson, you should be able to:

  • Design an eval before writing a prompt
  • Build a small harness with slices and a gate
  • Validate an LLM judge
  • Explain a 95% score that users hate

Part 3In detail

Step by step

  1. List failure modes first. Include those the spec missed: attacks, other languages, ambiguous cases, and errors with uneven costs.
  2. Build the set from real traffic. Refresh it regularly. Add every production failure.
  3. Write a rubric two people would agree on. If two graders disagree, the rubric isn't ready.
  4. Choose graders. Code checks where possible, a judge where necessary, and humans to calibrate.
  5. Validate any LLM judge. Compare it with human labels using kappa, not just raw agreement.
  6. Gate releases on slices. Block on any slice that drops, even if the average rises.

How many cases do you need?

Fewer than you'd fear to start, and more than you'd like to decide. 18 of 20 passing sounds like 90%, but the interval reaches down to about 70%. Use small sets to find problems, and larger ones before you make a close call between versions.

The code, explained

This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.

Running an eval

A loop over labelled cases, counting passes per slice.

EvalHarness.tsTypeScript
1export async function runEval<I, O>(cases: EvalCase<I>[], system: (input: I) => Promise<O>, grade: Grader<I, O>): Promise<EvalReport> {2  const report: EvalReport = { total: 0, passed: 0, byTag: {}, failures: [] };3  for (const c of cases) {4    const tag = (report.byTag[c.tag] ??= { total: 0, passed: 0 });5    report.total++;6    tag.total++;7    let ok = false;8    try {9      ok = grade(c.input, await system(c.input));10    } catch {11      ok = false; // a crash is a failed case, not a broken run12    }13    if (ok) {14      report.passed++;15      tag.passed++;16    } else {17      report.failures.push(c.id);18    }19  }20  return report;21}
21 linesUTF-8TypeScript
report.byTag[c.tag]
Results per slice are the point. One overall number hides too much.
ok = false; // a crash
One bad case shouldn't end the run. It counts as a failure and you move on.
report.failures.push(c.id)
Keep the ids. Reading failures is where the real learning happens.

Tested by npm run test:modules.

The release gate

Decides whether a new version may ship, and it checks each slice.

EvalHarness.tsTypeScript
1/** Checks every slice, because a better average can hide a slice that got much worse. */2export function releaseGate(base: EvalReport, next: EvalReport, maxDrop: number): { ship: boolean; blockers: string[] } {3  const rate = (r: { passed: number; total: number }) => (r.total ? r.passed / r.total : 0);4  const blockers: string[] = [];5  if (rate(next) < rate(base) - maxDrop) blockers.push("overall");6  for (const [tag, b] of Object.entries(base.byTag)) {7    const n = next.byTag[tag];8    if (!n) blockers.push(`${tag}: cases missing`);9    else if (rate(n) < rate(b) - maxDrop) blockers.push(tag);10  }11  return { ship: blockers.length === 0, blockers };12}
12 linesUTF-8TypeScript
rate(next) < rate(base) - maxDrop
The overall check, with a little room for noise.
for (const [tag, b] of Object.entries(base.byTag))
A slice check. In our test, the average rises from 90% to 93% while billing falls from 90% to 50%, and this blocks it.
cases missing
Deleting the hard cases must not count as a pass.

Tested by npm run test:modules.

How sure can you be?

A small eval set gives a wide range. This says how wide.

EvalHarness.tsTypeScript
1/** The honest range for a pass rate. 18 of 20 is 90%, but the data can't rule out about 70%. */2export function wilsonInterval(passed: number, total: number, z = 1.96): [number, number] {3  if (total === 0) return [0, 1];4  const p = passed / total;5  const d = 1 + (z * z) / total;6  const mid = p + (z * z) / (2 * total);7  const spread = z * Math.sqrt((p * (1 - p)) / total + (z * z) / (4 * total * total));8  return [(mid - spread) / d, (mid + spread) / d];9}
9 linesUTF-8TypeScript
z = 1.96
The value for a 95% interval.
(z * z) / (2 * total)
Pulls the estimate toward the middle, which keeps small samples honest.

Tested by npm run test:modules.

Checking an LLM judge

Raw agreement flatters a judge when most answers are good. Kappa takes out the agreement you'd get by luck.

EvalHarness.tsTypeScript
1/** Agreement after removing what chance alone would give. Check this before an LLM judge gates a release. */2export function cohenKappa(human: boolean[], judge: boolean[]): number {3  if (human.length !== judge.length || human.length === 0) throw new Error("need two equal, non-empty label lists");4  const n = human.length;5  const agree = human.filter((h, i) => h === judge[i]).length / n;6  const humanYes = human.filter(Boolean).length / n;7  const judgeYes = judge.filter(Boolean).length / n;8  const chance = humanYes * judgeYes + (1 - humanYes) * (1 - judgeYes);9  return chance === 1 ? 1 : (agree - chance) / (1 - chance);10}
10 linesUTF-8TypeScript
const chance
How often two graders would agree just from how often each says yes.
(agree - chance) / (1 - chance)
0 means no better than luck, and 1 means perfect.

Tested by npm run test:modules.

What can go wrong

A stale golden set
It stops matching what users ask, and the score becomes meaningless.
An unvalidated judge
It may pass everything, and you'd never know.
Only an average
In our test, the average rises from 90% to 93% while billing falls from 90% to 50%.
Mixing up model evals and system evals
A good retrieval score isn't a good product. Measure each layer.

Part 4Questions and answers

Answer out loud first, then open each one.

95% pass rate, and users complain. Name three possible causes.
The set no longer matches real traffic, the grader is too lenient, or the failures are concentrated in a slice that the average hides.
Why use kappa rather than raw agreement?
If 90% of answers are good, a judge that always says 'good' agrees 90% of the time and has learned nothing. Kappa removes the agreement expected by chance.
You must decide on a new model this week. What do you run?
The golden set on both versions, compared per slice, with cost and latency. Also a small live test if you can, and read the disagreements yourself.
How do production failures feed back?
Each becomes a labelled case in the set, so that it can't quietly return.

Part 5Going further

Drill: Build the take-home in 2 hours (120 min)

  1. Choose any small LLM feature, or fake one with a function.
  2. Write 30 cases, tagged by slice.
  3. Run the harness, then add a second version with one slice worse.
  4. Show the gate blocking it.
  5. Write one page: what you measured, what it missed, and what you'd do next.

Score yourself

Tick a line only if you did it without prompting.

0 of 6 Repeat the drill, and aim for the lines you missed.

Keep the fundamentals fresh with the critical basics.

Who says so

Checked in November 2026. Vendors and individuals often report what candidates told them, so treat their details as reports.

Next: the last round, where there's no code at all, and a customer says your bot feels dumb.

Next: Customer round →

← The agent design round

AI Engineer · AI interview rounds