/Interview

Build your own AI interviewer · Lesson 3 of 4 · 15 min

Build your coach: scoring you can trust

Study with AI

Part 1Basics

At the end, the coach scores you. This is where bad output does real harm: a kind, invented score sends you into an interview overconfident.

Three layers protect you. Ask for a clear format. Check everything in code. And require evidence: each score must quote your own words, and the code confirms that those words really appear in the transcript. A model can't quote something you never said and get away with it.

When the check fails, send the reason back and try once more. If it still fails, show no score. Admitting there's no result beats inventing one.

Words you need

Structured output
Model output in a fixed format, such as JSON. APIs can often enforce the shape, but you still check the meaning.
Validation
Code that checks the output before anyone uses it.
Grounding
Tying a claim to a source you can check.

A worked example

Meera's coach gives her a 4 for verification, with the evidence 'wrote thorough unit tests'. She never said that. The check rejects it, and the retry comes back honest: a 2, quoting 'I think that should be fine'. It's unpleasant to read, and it's exactly what she needed.

Part 2The question

How do you get reliable structured output from a model?

By the end of this lesson, you should be able to:

  • Validate model output
  • Retry with a useful reason
  • Fail honestly

Part 3In detail

Step by step

  1. Define the shape. Four competencies, each with a score and evidence.
  2. Parse it safely. Bad JSON is normal. Handle it.
  3. Check the meaning. Score range, and evidence that's really in the transcript.
  4. Retry once with the reason. Then stop.
  5. Keep the raw output. You'll need it for debugging and evals.

Why check evidence?

Schema enforcement guarantees the shape, and says nothing about truth. The evidence check is a small example of the most useful habit in AI engineering: find something you can verify in code.

The code, explained

This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.

Checking the score

Never trust a model's structured output until you've checked it.

Scorer.tsTypeScript
1/** Returns what's wrong in words, so the same text can go back to the model on a retry. */2export function parseScorecard(raw: string, transcript: string): { card: Scorecard } | { error: string } {3  let data: unknown;4  try {5    data = JSON.parse(raw);6  } catch {7    return { error: "Reply with JSON only." };8  }9  const obj = data as Record<string, ScoreLine | undefined>;10  for (const c of COMPETENCIES) {11    const line = obj?.[c];12    if (!line || !Number.isInteger(line.score) || line.score < 1 || line.score > 5) return { error: `${c}.score must be a whole number from 1 to 5.` };13    if (typeof line.evidence !== "string" || line.evidence.length < 10) return { error: `${c}.evidence must quote the transcript.` };14    if (!transcript.includes(line.evidence)) return { error: `${c}.evidence isn't in the transcript. Copy the candidate's exact words.` };15  }16  return { card: obj as unknown as Scorecard };17}
17 linesUTF-8TypeScript
JSON.parse(raw)
The first check: is it JSON at all?
Number.isInteger(line.score)
Keeps scores to whole numbers from 1 to 5.
transcript.includes(line.evidence)
The most useful line here. Evidence must be real words from the transcript, so a model can't invent a reason for a score.

Tested by npm run test:modules.

One retry, then nothing

Give the model one chance to fix it, and tell it what was wrong.

Scorer.tsTypeScript
1export type Complete = (prompt: string) => Promise<string>;2 3/** One retry with the reason. After that, show no score: a made-up score is worse than none. */4export async function scoreInterview(complete: Complete, prompt: string, transcript: string): Promise<Scorecard | null> {5  let ask = prompt;6  for (let attempt = 0; attempt < 2; attempt++) {7    const result = parseScorecard(await complete(ask), transcript);8    if ("card" in result) return result.card;9    ask = `${prompt}\n\nYour last reply was rejected: ${result.error}`;10  }11  return null;12}
12 linesUTF-8TypeScript
attempt < 2
A strict limit. Endless retries cost money and rarely help.
Your last reply was rejected
Sending the reason back gives the retry something to fix.
return null
Showing no score is better than showing a made-up one.

Tested by npm run test:modules.

What can go wrong

Trusting valid JSON
Correct shape, invented content.
Endless retries
They cost money and rarely help. One, then stop.
A default score on failure
A made-up number looks just like a real one.

Part 4Questions and answers

Answer out loud first, then open each one.

What does schema enforcement not protect you from?
Wrong or invented content. The shape can be perfect, and the facts false.
Why send the error back on a retry?
So the model knows what to fix. Without the reason, it's likely to make the same mistake.
What should the coach show if scoring fails twice?
No score, and an honest message. A fake number is worse than none, since you'd act on it.

Part 5Going further

Drill: Attack the scorer (15 min)

  1. Write 5 bad replies: invalid JSON, a score of 6, a missing field, invented evidence, and evidence too short to mean anything.
  2. Confirm each is rejected for the right reason.
  3. Find one more way to fool it, and add a check.

Keep the fundamentals fresh with the critical basics.

Next: prove the coach works, then turn it into your best interview story.

Next: Evals and your story →

← Build your coach: the prompt and the model call

AI Engineer · Build your own AI interviewer