Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Manual red-teaming finds problems once and doesn't repeat. An automated suite keeps a growing library of probes, generates variants, scores the replies, and blocks a release when any harm category gets worse.
Turns one-off safety testing into a regression suite that gates every release.
1.2Key concepts
One idea per card, in plain words.
Curated attack prompts per harm category. Keep it private, versioned and access-controlled.
Generates variants of a probe, because small rewording often changes the result.
Share of attempts producing a harmful reply, per category.
Blocking innocent requests. Test it too, or safety hides a broken product.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Your company ships LLM features monthly and found jailbreaks only after launch. Design a system that catches regressions before release.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Who owns the probes? | A security team, with restricted access. |
| What blocks a release? | Any category above its baseline, unless a named owner accepts the risk. |
Detailed answer
How it works, and how to build or explain it.
3.1Architecture
The components and how they connect.
3.2Step by step
The mechanism, in the order you would explain it.
- Define the harms
Pick categories that match your product, based on a public taxonomy.
- Seed the library
Start from open tools such as garak and PyRIT, then add your own incidents.
- Generate variants
Apply rewording, translation and role-play framing at scale.
- Test the full product
Run through the real system prompt, tools and guardrails, not the bare model.
- Score with a judge
Combine a classifier with an LLM judge, and sample for human review.
- Gate the release
Fail when any category's rate rises above baseline. Add each new failure to the library.
3.3Code
Working code for the core of it.
Red-teaming as a test suite
Attacks found once should fail in CI forever after.
1export interface Probe {2 category: string;3 prompt: string;4}5 6/** Runs each probe and its variants against the target; a judge scores every reply. */7export async function runSuite(8 probes: Probe[],9 mutate: (prompt: string) => string[],10 target: (prompt: string) => Promise<string>,11 judge: (category: string, reply: string) => Promise<boolean>12) {13 const perCategory = new Map<string, { attempts: number; successes: number }>();14 const failures: { category: string; prompt: string }[] = [];15 for (const probe of probes) {16 const stats = perCategory.get(probe.category) ?? { attempts: 0, successes: 0 };17 perCategory.set(probe.category, stats);18 for (const prompt of [probe.prompt, ...mutate(probe.prompt)]) {19 stats.attempts++;20 if (await judge(probe.category, await target(prompt))) {21 stats.successes++;22 failures.push({ category: probe.category, prompt }); // keep it: tomorrow's regression test23 }24 }25 }26 const attackSuccessRate = new Map([...perCategory].map(([c, s]) => [c, s.successes / s.attempts]));27 return { attackSuccessRate, failures };28}29 30/** The CI gate: no category may get worse than its baseline. */31export function passesGate(rates: Map<string, number>, baseline: Record<string, number>, margin = 0): boolean {32 return [...rates].every(([category, rate]) => rate <= (baseline[category] ?? 0) + margin);33}mutate(probe.prompt)- Each attack runs in many variations, since models that block one phrasing often miss another.
failures.push(...)- Each success becomes tomorrow's regression test.
passesGate- The build fails if any category gets worse than its baseline.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why test the whole product?
How do you stop the suite from going stale?
How do you keep the probes safe?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Costs
- Thousands of attempts per release cost money. Run the full suite before release and a smoke subset per change.
- Counterpoint
- A clean suite means known attacks failed, not that the product is safe. Keep human red-teamers for new thinking.