Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
AI adoption questions test principal engineers: how AI changes the way teams build, and how you roll it out responsibly. Cover tools, workflow, guardrails, measurement and culture, not just which assistant to buy.
- Reading and explaining unfamiliar code
- Writing tests, migrations and boilerplate
- Refactors with clear patterns across many files
- First drafts of RFCs, runbooks and incident timelines
- Reviewing diffs for common mistakes
- Investigating failures with logs and traces
- Security-sensitive code without expert review
- Architecture decisions without human ownership
- Merging generated code nobody understands
- Sending proprietary code to tools without an enterprise agreement
1.2Key concepts
Ideas this deep dive relies on, in one sentence each.
The last server sequence number a client has seen; pulls ask for changes after it.
Model Context Protocol: a standard for exposing tools and data to AI applications.
1.3Rolling out AI coding tools to 150 engineers
A concrete example first, so the mechanics have something to attach to.
Leadership wants AI tools for 150 mobile and backend engineers. You start with a four-week pilot in two teams: one mobile, one backend, with an enterprise plan that keeps code private and disables training on company data.
Engineers use an IDE assistant for completion and chat, and an agentic coding tool such as Claude Code for multi-file changes: it reads the repo, runs Gradle and tests, and proposes diffs that engineers review. A repository-level instructions file captures conventions: module boundaries, test commands and banned APIs.
CI adds an AI review pass that comments on likely bugs and missing tests; humans still approve. Test generation targets the sync engine, where coverage was weakest.
You measure before and after: PR cycle time, change failure rate, escaped defects, and a short developer survey. The team shares results openly, including where the tools did not help, before expanding to all teams with a paved-road setup.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- How would you roll out AI coding tools across teams?
- How do you measure whether AI helps?
- What would you never let an AI agent do?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1AI across the delivery pipeline
The parts and how they connect.
- Context is the multiplier
- Assistants perform far better with repository instructions, architecture docs and access to internal docs through MCP. Treat that context as maintained infrastructure.
- Agents for multi-step changes
- Agentic tools can run builds and tests and iterate on failures. Keep them in sandboxes with the developer approving commands and diffs.
- Human ownership
- Every merged line has a human owner who understands it. AI review supplements, never replaces, human approval.
- Same quality gates
- Generated code passes the same lint, tests, security scanning and dependency checks as any other code.
3.2Tool categories
| Name | Meaning |
|---|---|
| Completion | Inline suggestions as you type. |
| Chat in IDE | Explain, refactor, generate with selected context. |
| Coding agents | Read the repo, run commands, edit files, iterate on test failures. |
| AI code review | Automated comments on pull requests. |
| Ops assistants | Query logs, metrics and traces in natural language. |
3.3An AI-assisted change
One run, step by step.
- The engineer defines done and owns the result.
- CI and human review are unchanged; AI shortens the loop before them.
3.4Code
Working code for the core of it.
Repository instructions file (e.g. CLAUDE.md or AGENTS.md)
1# Mobile app conventions2- Kotlin, Jetpack Compose, Hilt. Feature modules: :feature:<name>:api and :impl.3- Never depend on another feature's :impl module.4- Database access only through Room DAOs in :core:db; every write in withTransaction.5- Run before proposing a change: ./gradlew :app:lintDebug testDebugUnitTest6- Sync engine changes need a fault-injection test in :core:sync:test.7- Do not add dependencies without asking.AI review step in CI (GitHub Actions sketch)
1name: ai-review2on: pull_request3jobs:4 review:5 runs-on: ubuntu-latest6 permissions: { pull-requests: write, contents: read }7 steps:8 - uses: actions/checkout@v49 - name: AI pre-review (comments only, never approves)10 run: ./scripts/ai-review.sh --diff origin/main...HEAD --rules .review-rules.md3.5Tested code, explained
A working implementation, then what each part does and why.
Checking an LLM judge
A judge you haven't measured is an opinion. These two functions make it a measurement.
1/** Cohen's kappa: agreement between two raters, beyond what chance would produce. */2export function cohenKappa(a: string[], b: string[]): number {3 const n = a.length;4 const labels = new Set([...a, ...b]);5 const observed = a.filter((x, i) => x === b[i]).length / n;6 let chance = 0;7 for (const l of labels) chance += (a.filter((x) => x === l).length / n) * (b.filter((x) => x === l).length / n);8 return chance === 1 ? 1 : (observed - chance) / (1 - chance);9}10 11export type Verdict = "first" | "second" | "tie";12 13/** Removes position bias: ask twice with the answers swapped; a verdict counts only if both agree. */14export async function debiasedVerdict(judge: (x: string, y: string) => Promise<Verdict>, a: string, b: string): Promise<"a" | "b" | "tie"> {15 const [one, two] = await Promise.all([judge(a, b), judge(b, a)]);16 if (one === "first" && two === "second") return "a";17 if (one === "second" && two === "first") return "b";18 return "tie";19}cohenKappa- Agreement with human labels, beyond what chance would give.
Promise.all([judge(a, b), judge(b, a)])- Judges favour one position, so ask both ways and count only agreement.
Tested by npm run test:modules.
Red-teaming as a test suite
Attacks found once should fail in CI forever after.
1export interface Probe {2 category: string;3 prompt: string;4}5 6/** Runs each probe and its variants against the target; a judge scores every reply. */7export async function runSuite(8 probes: Probe[],9 mutate: (prompt: string) => string[],10 target: (prompt: string) => Promise<string>,11 judge: (category: string, reply: string) => Promise<boolean>12) {13 const perCategory = new Map<string, { attempts: number; successes: number }>();14 const failures: { category: string; prompt: string }[] = [];15 for (const probe of probes) {16 const stats = perCategory.get(probe.category) ?? { attempts: 0, successes: 0 };17 perCategory.set(probe.category, stats);18 for (const prompt of [probe.prompt, ...mutate(probe.prompt)]) {19 stats.attempts++;20 if (await judge(probe.category, await target(prompt))) {21 stats.successes++;22 failures.push({ category: probe.category, prompt }); // keep it: tomorrow's regression test23 }24 }25 }26 const attackSuccessRate = new Map([...perCategory].map(([c, s]) => [c, s.successes / s.attempts]));27 return { attackSuccessRate, failures };28}29 30/** The CI gate: no category may get worse than its baseline. */31export function passesGate(rates: Map<string, number>, baseline: Record<string, number>, margin = 0): boolean {32 return [...rates].every(([category, rate]) => rate <= (baseline[category] ?? 0) + margin);33}mutate(probe.prompt)- Each attack runs in many variations, since models that block one phrasing often miss another.
failures.push(...)- Each success becomes tomorrow's regression test.
passesGate- The build fails if any category gets worse than its baseline.
Tested by npm run test:modules.
3.6Guardrails that make it safe
Data and security
- Enterprise agreements with no training on your code and clear retention terms.
- Secrets scanning before any code leaves the machine; block tools on regulated repositories if needed.
Quality
- Same CI gates for all code; stricter review on security-sensitive paths.
- Track escaped defects and change failure rate for AI-heavy changes.
Culture
- Share prompts, instructions files and good workflows in a guild.
- Make it optional at first; publish results honestly.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
How would you roll out AI coding tools across teams?
How do you measure whether AI helps?
What would you never let an AI agent do?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Tools
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| IDE assistants | GitHub Copilot · Cursor · JetBrains AI · others | Pick by IDE coverage, enterprise controls and model choice. |
| Coding agents | Claude Code · other terminal and IDE agents | For multi-file changes, test fixing and codebase exploration. |
| Review | AI review bots in CI | Comments only; humans approve. |
| Context | Instructions files · MCP servers for docs, tickets, logs | Keep them versioned with the code. |
5.2Measuring impact honestly
Show detailsHide details
| Metric | Why it matters |
|---|---|
| PR cycle time and lead time | Delivery speed (DORA). |
| Change failure rate and escaped defects | Quality must not drop. |
| Review load per reviewer | AI can increase PR volume; watch reviewer burnout. |
| Developer survey | Perceived productivity and where it helps. |
5.3Failure modes
Show detailsHide details
| Case | How it is handled |
|---|---|
| More code, more review burden | Smaller PRs, AI pre-review, clear ownership. |
| Subtle bugs in plausible code | Tests first, fault injection for critical modules, careful review. |
| Licensing and IP concerns | Tools with code-reference filters and enterprise terms. |
| Skills atrophy in juniors | Pairing, explain-before-merge, and code reading practice. |