AI interview rounds · Lesson 8 of 8 · 15 min
The customer conversation round
Part 1Basics
Some AI teams role-play a customer conversation, because their engineers actually talk to customers. One write-up lists weakness here as a common reason for rejection. Even without this round, the skill helps in every behavioural interview.
A complaint is a symptom, and the words are often vague. Your first job is to get a concrete example. Your second is to work out which layer failed. Your third is to say honestly what happens next.
Words you need
- Reproduction
- One real example you can run again. Without one, you're guessing.
- Trace
- The record of what the system did for one request.
- Severity
- How much harm, to how many people. It decides what happens first.
A worked example
'It feels dumb,' the customer says. Meera doesn't defend the product or promise a fix. She asks for an example from yesterday. The customer shares one, and she reads the trace: the right document was never retrieved, since it was uploaded on Tuesday and the index last ran on Monday. She explains that in plain language, gives a short-term workaround, and says when she'll follow up. She doesn't mention embeddings at all.
Part 2The question
A customer says your bot feels dumb. What do you do in the next ten minutes?
By the end of this lesson, you should be able to:
- Turn a vague complaint into one example
- Find the failing layer from a trace
- Explain it without jargon, and promise only what you can do
Part 3In detail
Step by step
- Listen, and ask for an example. 'Can you show me one from this week?' Vague complaints become specific.
- Ask what they expected. The gap between expected and actual is the bug.
- Read the trace, in order. Tool errors, then retrieval, then citations, then the answer.
- Say what you found, in their words. No jargon. Say what happened and what it means for them.
- Commit to a next step. A workaround now, a date for the fix, or an honest 'I don't know yet, and I'll know by Friday.'
- Count it. Is this one customer, or many? Check across traces before you prioritise.
Prompts to copy
Practise with a difficult customer
Answer out loud, or type quickly. Don't polish.
Role-play a frustrated customer of an AI support assistant. You say the bot 'feels dumb'. You're busy, not technical, and vague at first. Only give a concrete example if I ask for one. Don't help me. After 8 exchanges, step out of character and tell me: whether I got a real example, whether I used jargon, whether I promised something I couldn't deliver, and what I should have asked.
The code, explained
This is real code, and our tests run it, including the buggy versions, to prove the bugs are real.
From complaint to cause
Walks the stages in order, and reports the first one that failed.
1/** The order matters: an early failure explains the later ones, so report the first one you find. */2export function diagnose(t: Trace, latencyLimitMs: number): Finding {3 if (t.toolErrors.length > 0) return "tool-failed";4 if (!t.neededDocs.some((d) => t.retrievedDocs.includes(d))) return "retrieval-missed";5 if (t.citedDocs.some((d) => !t.retrievedDocs.includes(d))) return "made-up-citation";6 if (!t.answerUsedNeededDoc) return "context-ignored";7 if (t.latencyMs > latencyLimitMs) return "too-slow";8 return "no-fault-found";9}t.toolErrors.length > 0- Check the plumbing first. A failed tool explains everything after it.
!t.neededDocs.some- The right text never arrived, so no prompt change will help.
t.citedDocs.some- It cites something it never retrieved, which means it made up the citation.
!t.answerUsedNeededDoc- The text was there and the model ignored it. Now the prompt is worth changing.
Tested by npm run test:modules.
From one complaint to a priority
Run the same check across many traces to see what to fix first.
1/** One complaint is a story. Counting findings across traces tells you what to fix first. */2export function rankFindings(traces: Trace[], latencyLimitMs: number): [Finding, number][] {3 const counts = new Map<Finding, number>();4 for (const t of traces) {5 const f = diagnose(t, latencyLimitMs);6 counts.set(f, (counts.get(f) ?? 0) + 1);7 }8 return [...counts].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]));9}counts.set(f, (counts.get(f) ?? 0) + 1)- One customer's story becomes a measurement across all of them.
Tested by npm run test:modules.
What can go wrong
- Defending the product
- The customer doesn't care why it's hard. Listen first.
- Jargon
- Saying 'our recall dropped' helps nobody. Say 'it couldn't find your new document.'
- Promising a date you can't keep
- Say what you know now, and when you'll know more.
Part 4Questions and answers
Answer out loud first, then open each one.
Why check tool errors first in a trace?
The customer's example works fine when you try it. What now?
What makes a good next step?
Part 5Going further
Drill: Eight exchanges (15 min)
- Run the difficult customer prompt.
- Read its feedback.
- Run it once more, fixing the top issue.
Keep the fundamentals fresh with the critical basics.
Who says so
Checked in November 2026. Vendors and individuals often report what candidates told them, so treat their details as reports.
- What applied-AI engineer interviews test Individual, July 2026
- Stop testing implementation, start probing eval design Individual, April 2026
That's every round. Next path: how to put it all together into a plan, with AI as your training partner.