/Interview

AI interview rounds · Lesson 8 of 8 · 15 min

The customer conversation round

Study with AI

Part 1Basics

Some AI teams role-play a customer conversation, because their engineers actually talk to customers. One write-up lists weakness here as a common reason for rejection. Even without this round, the skill helps in every behavioural interview.

A complaint is a symptom, and the words are often vague. Your first job is to get a concrete example. Your second is to work out which layer failed. Your third is to say honestly what happens next.

Words you need

Reproduction
One real example you can run again. Without one, you're guessing.
Trace
The record of what the system did for one request.
Severity
How much harm, to how many people. It decides what happens first.

A worked example

'It feels dumb,' the customer says. Meera doesn't defend the product or promise a fix. She asks for an example from yesterday. The customer shares one, and she reads the trace: the right document was never retrieved, since it was uploaded on Tuesday and the index last ran on Monday. She explains that in plain language, gives a short-term workaround, and says when she'll follow up. She doesn't mention embeddings at all.

Part 2The question

A customer says your bot feels dumb. What do you do in the next ten minutes?

By the end of this lesson, you should be able to:

  • Turn a vague complaint into one example
  • Find the failing layer from a trace
  • Explain it without jargon, and promise only what you can do

Part 3In detail

Step by step

  1. Listen, and ask for an example. 'Can you show me one from this week?' Vague complaints become specific.
  2. Ask what they expected. The gap between expected and actual is the bug.
  3. Read the trace, in order. Tool errors, then retrieval, then citations, then the answer.
  4. Say what you found, in their words. No jargon. Say what happened and what it means for them.
  5. Commit to a next step. A workaround now, a date for the fix, or an honest 'I don't know yet, and I'll know by Friday.'
  6. Count it. Is this one customer, or many? Check across traces before you prioritise.

Prompts to copy

Practise with a difficult customer

Answer out loud, or type quickly. Don't polish.

Role-play a frustrated customer of an AI support assistant. You say the bot 'feels dumb'. You're busy, not technical, and vague at first. Only give a concrete example if I ask for one. Don't help me. After 8 exchanges, step out of character and tell me: whether I got a real example, whether I used jargon, whether I promised something I couldn't deliver, and what I should have asked.

The code, explained

This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.

From complaint to cause

Walks the stages in order, and reports the first one that failed.

TraceTriage.tsTypeScript
1/** The order matters: an early failure explains the later ones, so report the first one you find. */2export function diagnose(t: Trace, latencyLimitMs: number): Finding {3  if (t.toolErrors.length > 0) return "tool-failed";4  if (!t.neededDocs.some((d) => t.retrievedDocs.includes(d))) return "retrieval-missed";5  if (t.citedDocs.some((d) => !t.retrievedDocs.includes(d))) return "made-up-citation";6  if (!t.answerUsedNeededDoc) return "context-ignored";7  if (t.latencyMs > latencyLimitMs) return "too-slow";8  return "no-fault-found";9}
9 linesUTF-8TypeScript
t.toolErrors.length > 0
Check the plumbing first. A failed tool explains everything after it.
!t.neededDocs.some
The right text never arrived, so no prompt change will help.
t.citedDocs.some
It cites something it never retrieved, which means it made up the citation.
!t.answerUsedNeededDoc
The text was there and the model ignored it. Now the prompt is worth changing.

Tested by npm run test:modules.

From one complaint to a priority

Run the same check across many traces to see what to fix first.

TraceTriage.tsTypeScript
1/** One complaint is a story. Counting findings across traces tells you what to fix first. */2export function rankFindings(traces: Trace[], latencyLimitMs: number): [Finding, number][] {3  const counts = new Map<Finding, number>();4  for (const t of traces) {5    const f = diagnose(t, latencyLimitMs);6    counts.set(f, (counts.get(f) ?? 0) + 1);7  }8  return [...counts].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));9}
9 linesUTF-8TypeScript
counts.set(f, (counts.get(f) ?? 0) + 1)
One customer's story becomes a measurement across all of them.

Tested by npm run test:modules.

What can go wrong

Defending the product
The customer doesn't care why it's hard. Listen first.
Jargon
Saying 'our recall dropped' helps nobody. Say 'it couldn't find your new document.'
Promising a date you can't keep
Say what you know now, and when you'll know more.

Part 4Questions and answers

Answer out loud first, then open each one.

Why check tool errors first in a trace?
An early failure explains the later ones. If the search tool failed, the bad answer is a symptom.
The customer's example works fine when you try it. What now?
Ask what they saw, and look at their actual trace. Differences in documents, permissions, or timing are common causes, and it may be a real intermittent failure.
What makes a good next step?
It's specific, it's something you control, and it has a date. 'I'll look into it' isn't one.

Part 5Going further

Drill: Eight exchanges (15 min)

  1. Run the difficult customer prompt.
  2. Read its feedback.
  3. Run it once more, fixing the top issue.

Keep the fundamentals fresh with the critical basics.

Who says so

Checked in November 2026. Vendors and individuals often report what candidates told them, so treat their details as reports.

That's every round. Next path: how to put it all together into a plan, with AI as your training partner.

Next path: Playbook →

← The evals round

AI Engineer · AI interview rounds