Build your own AI interviewer · Lesson 4 of 4 · 20 min
Build your coach: evals, cost and your story
Part 1Basics
Your coach gives scores. Are they any good? Score 20 of your mocks yourself, then compare. You're looking at two things: how often it's within a point of you, and whether it's systematically too kind.
Once you have those numbers, you have something few candidates have: a real AI project, with a real failure, a real measurement, and a real fix. 'Tell me about something you built with AI' becomes your strongest answer.
Words you need
- Bias
- A consistent lean in one direction. Here, scoring you too high.
- Calibration
- Comparing a judge with human labels and adjusting it.
- STAR
- Situation, task, action, result. A common shape for behavioural answers.
A worked example
Meera labels 20 scores herself. Within-one agreement is 85%, but the bias is +0.8: the coach is too kind. She adds 'scores of 4 and 5 need two pieces of evidence' to the prompt, and the bias falls to +0.3. Her interview story is now one sentence long and concrete: 'My coach scored me about 0.8 points too high, I found out by labelling 20 sessions, and one rule cut that to 0.3.'
Part 2The question
How do you prove it works, and how do you talk about it in an interview?
By the end of this lesson, you should be able to:
- Evaluate your coach
- Work out what it costs
- Tell the project as a true, specific story
Part 3In detail
Step by step
- Save every session. Transcripts are your data.
- Label 20 by hand. Before you look at the coach's score.
- Compare. Within-one agreement, and bias.
- Change one thing and rerun. That's a real iteration, with a measurement.
- Work out the cost per mock. Tokens times prices. Note how much caching saves.
- Write your story. One failure, one measurement, one fix. Only true numbers.
The story, in four lines
Why: 'I wanted honest practice for an AI interview.' What: 'A mock interviewer. Code owns the phases, and the model writes the words.' What went wrong: 'It scored too kindly. I measured that against my own labels.' What changed: 'One rule moved the bias from this to that, and here's what I'd do next.'
Expect these follow-ups
Why not a bigger model? How would it handle ten thousand users? How would you stop abuse? What if the labels are wrong, given that they're all yours? Having real answers to these is what the project gets you. Your own labels are a real weakness, so say so before they do.
Use your own numbers
Meera's figures are made up for this example. Only ever quote numbers you actually measured. A smaller true number will survive follow-up questions, and an invented one won't.
The code, explained
This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.
Does it score like a person?
Score some of your mocks by hand, then compare.
1export function agreement(rows: LabelledScore[]) {2 if (rows.length === 0) throw new Error("no labelled scores yet");3 const share = (test: (r: LabelledScore) => boolean) => rows.filter(test).length / rows.length;4 return {5 exact: share((r) => r.human === r.model),6 withinOne: share((r) => Math.abs(r.human - r.model) <= 1),7 /** above 0 means the coach flatters you */8 bias: rows.reduce((a, r) => a + (r.model - r.human), 0) / rows.length,9 };10}Math.abs(r.human - r.model) <= 1- People don't agree exactly either, so one point apart is a fair test.
bias- The average difference, with its sign. Positive means it's too kind.
Tested by npm run test:modules.
When to trust your coach
The same idea as a release gate, for a project of your own.
1/** A coach that flatters is the failure that hurts you most, so bias gets its own limit. */2export function coachIsTrustworthy(rows: LabelledScore[]): { ok: boolean; reasons: string[] } {3 const a = agreement(rows);4 const reasons: string[] = [];5 if (rows.length < 20) reasons.push(`only ${rows.length} labelled scores; label at least 20`);6 if (a.withinOne < 0.8) reasons.push(`within-one agreement is ${Math.round(a.withinOne * 100)}%, below 80%`);7 if (a.bias > 0.5) reasons.push(`scores run ${a.bias.toFixed(1)} points too kind`);8 return { ok: reasons.length === 0, reasons };9}rows.length < 20- Too few labels can't tell you much.
a.bias > 0.5- A flattering coach sends you to the interview overconfident, so it gets its own limit.
Tested by npm run test:modules.
What can go wrong
- Looking at the coach's score before labelling
- You'll anchor on it. Label blind.
- Only one person's labels
- It's a real limit. Say so, and ask a friend to label some.
- Overselling a side project
- State honestly who used it: probably just you. That's fine.
Part 4Questions and answers
Answer out loud first, then open each one.
Why does bias get its own limit?
What makes this a good interview story?
An interviewer asks how it would scale. What's a good first answer?
Part 5Going further
Drill: Write your story (30 min)
- Build as much of the coach as you can in a weekend.
- Label at least 10 sessions.
- Write your four lines with true numbers.
- Have the cross-examination prompt attack it, and fix what it finds.
Score yourself
Tick a line only if you did it without prompting.
0 of 4 Repeat the drill, and aim for the lines you missed.
Keep the fundamentals fresh with the critical basics.
You've finished the paths. Keep the critical basics in your daily review, and good luck.