/Interview

Build your own AI interviewer · Lesson 4 of 4 · 20 min

Build your coach: evals, cost and your story

Study with AI

Part 1Basics

Your coach gives scores. Are they any good? Score 20 of your mocks yourself, then compare. You're looking at two things: how often it's within a point of you, and whether it's systematically too kind.

Once you have those numbers, you have something few candidates have: a real AI project, with a real failure, a real measurement, and a real fix. 'Tell me about something you built with AI' becomes your strongest answer.

Words you need

Bias
A consistent lean in one direction. Here, scoring you too high.
Calibration
Comparing a judge with human labels and adjusting it.
STAR
Situation, task, action, result. A common shape for behavioural answers.

A worked example

Meera labels 20 scores herself. Within-one agreement is 85%, but the bias is +0.8: the coach is too kind. She adds 'scores of 4 and 5 need two pieces of evidence' to the prompt, and the bias falls to +0.3. Her interview story is now one sentence long and concrete: 'My coach scored me about 0.8 points too high, I found out by labelling 20 sessions, and one rule cut that to 0.3.'

Part 2The question

How do you prove it works, and how do you talk about it in an interview?

By the end of this lesson, you should be able to:

  • Evaluate your coach
  • Work out what it costs
  • Tell the project as a true, specific story

Part 3In detail

Step by step

  1. Save every session. Transcripts are your data.
  2. Label 20 by hand. Before you look at the coach's score.
  3. Compare. Within-one agreement, and bias.
  4. Change one thing and rerun. That's a real iteration, with a measurement.
  5. Work out the cost per mock. Tokens times prices. Note how much caching saves.
  6. Write your story. One failure, one measurement, one fix. Only true numbers.

The story, in four lines

Why: 'I wanted honest practice for an AI interview.' What: 'A mock interviewer. Code owns the phases, and the model writes the words.' What went wrong: 'It scored too kindly. I measured that against my own labels.' What changed: 'One rule moved the bias from this to that, and here's what I'd do next.'

Expect these follow-ups

Why not a bigger model? How would it handle ten thousand users? How would you stop abuse? What if the labels are wrong, given that they're all yours? Having real answers to these is what the project gets you. Your own labels are a real weakness, so say so before they do.

Use your own numbers

Meera's figures are made up for this example. Only ever quote numbers you actually measured. A smaller true number will survive follow-up questions, and an invented one won't.

The code, explained

This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.

Does it score like a person?

Score some of your mocks by hand, then compare.

CoachEval.tsTypeScript
1export function agreement(rows: LabelledScore[]) {2  if (rows.length === 0) throw new Error("no labelled scores yet");3  const share = (test: (r: LabelledScore) => boolean) => rows.filter(test).length / rows.length;4  return {5    exact: share((r) => r.human === r.model),6    withinOne: share((r) => Math.abs(r.human - r.model) <= 1),7    /** above 0 means the coach flatters you */8    bias: rows.reduce((a, r) => a + (r.model - r.human), 0) / rows.length,9  };10}
10 linesUTF-8TypeScript
Math.abs(r.human - r.model) <= 1
People don't agree exactly either, so one point apart is a fair test.
bias
The average difference, with its sign. Positive means it's too kind.

Tested by npm run test:modules.

When to trust your coach

The same idea as a release gate, for a project of your own.

CoachEval.tsTypeScript
1/** A coach that flatters is the failure that hurts you most, so bias gets its own limit. */2export function coachIsTrustworthy(rows: LabelledScore[]): { ok: boolean; reasons: string[] } {3  const a = agreement(rows);4  const reasons: string[] = [];5  if (rows.length < 20) reasons.push(`only ${rows.length} labelled scores; label at least 20`);6  if (a.withinOne < 0.8) reasons.push(`within-one agreement is ${Math.round(a.withinOne * 100)}%, below 80%`);7  if (a.bias > 0.5) reasons.push(`scores run ${a.bias.toFixed(1)} points too kind`);8  return { ok: reasons.length === 0, reasons };9}
9 linesUTF-8TypeScript
rows.length < 20
Too few labels can't tell you much.
a.bias > 0.5
A flattering coach sends you to the interview overconfident, so it gets its own limit.

Tested by npm run test:modules.

What can go wrong

Looking at the coach's score before labelling
You'll anchor on it. Label blind.
Only one person's labels
It's a real limit. Say so, and ask a friend to label some.
Overselling a side project
State honestly who used it: probably just you. That's fine.

Part 4Questions and answers

Answer out loud first, then open each one.

Why does bias get its own limit?
A coach that flatters is the failure that hurts most: you'd walk in overconfident. Agreement alone could hide that if every score is one point too high.
What makes this a good interview story?
It has a real failure, a measurement, a change, and a result, all of which you can defend with detail, since you did them.
An interviewer asks how it would scale. What's a good first answer?
Say what would break first, such as cost per session and API rate limits, then how you'd deal with it: caching, queues and limits per user. And say you haven't had to do it yet.

Part 5Going further

Drill: Write your story (30 min)

  1. Build as much of the coach as you can in a weekend.
  2. Label at least 10 sessions.
  3. Write your four lines with true numbers.
  4. Have the cross-examination prompt attack it, and fix what it finds.

Score yourself

Tick a line only if you did it without prompting.

0 of 4 Repeat the drill, and aim for the lines you missed.

Keep the fundamentals fresh with the critical basics.

You've finished the paths. Keep the critical basics in your daily review, and good luck.

Back to all paths

← Build your coach: scoring you can trust

AI Engineer · Build your own AI interviewer