/Learn
Modules · Evals and guardrails · System design

Automated Adversarial Red-Teaming Suites

PracticeSystem designTier 1 · Whole pictureCreate35 minEvals and guardrails

You will be able to: design an automated red-teaming pipeline with a release gate and a human review loop.

Source: Perez et al., 2022; NVIDIA garak; Microsoft PyRIT

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Manual red-teaming finds problems once and doesn't repeat. An automated suite keeps a growing library of probes, generates variants, scores the replies, and blocks a release when any harm category gets worse.

Why it matters

Turns one-off safety testing into a regression suite that gates every release.

1.2Key concepts

One idea per card, in plain words.

01Probe library

Curated attack prompts per harm category. Keep it private, versioned and access-controlled.

02Mutation

Generates variants of a probe, because small rewording often changes the result.

03Attack success rate

Share of attempts producing a harmful reply, per category.

04Over-refusal

Blocking innocent requests. Test it too, or safety hides a broken product.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Your company ships LLM features monthly and found jailbreaks only after launch. Design a system that catches regressions before release.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Who owns the probes?A security team, with restricted access.
What blocks a release?Any category above its baseline, unless a named owner accepts the risk.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Architecture

The components and how they connect.

Probe libraryversioned · per categoryVariant generatortemplates + LLMSuite runnerCI · rate limitedTargetprompt + model + toolsJudgeclassifier + LLM judgeRelease gatecompare with baselineHuman reviewnew failures → libraryattemptspromptreplyratessamplesnew probes
ClientServiceData storeAI component
Boxes are components; arrows show calls and data flow.

3.2Step by step

The mechanism, in the order you would explain it.

  1. Define the harms

    Pick categories that match your product, based on a public taxonomy.

  2. Seed the library

    Start from open tools such as garak and PyRIT, then add your own incidents.

  3. Generate variants

    Apply rewording, translation and role-play framing at scale.

  4. Test the full product

    Run through the real system prompt, tools and guardrails, not the bare model.

  5. Score with a judge

    Combine a classifier with an LLM judge, and sample for human review.

  6. Gate the release

    Fail when any category's rate rises above baseline. Add each new failure to the library.

3.3Code

Working code for the core of it.

Red-teaming as a test suite

Attacks found once should fail in CI forever after.

red-team-loop.tsTypeScript
1export interface Probe {2  category: string;3  prompt: string;4}5 6/** Runs each probe and its variants against the target; a judge scores every reply. */7export async function runSuite(8  probes: Probe[],9  mutate: (prompt: string) => string[],10  target: (prompt: string) => Promise<string>,11  judge: (category: string, reply: string) => Promise<boolean>12) {13  const perCategory = new Map<string, { attempts: number; successes: number }>();14  const failures: { category: string; prompt: string }[] = [];15  for (const probe of probes) {16    const stats = perCategory.get(probe.category) ?? { attempts: 0, successes: 0 };17    perCategory.set(probe.category, stats);18    for (const prompt of [probe.prompt, ...mutate(probe.prompt)]) {19      stats.attempts++;20      if (await judge(probe.category, await target(prompt))) {21        stats.successes++;22        failures.push({ category: probe.category, prompt }); // keep it: tomorrow's regression test23      }24    }25  }26  const attackSuccessRate = new Map([...perCategory].map(([c, s]) => [c, s.successes / s.attempts]));27  return { attackSuccessRate, failures };28}29 30/** The CI gate: no category may get worse than its baseline. */31export function passesGate(rates: Map<string, number>, baseline: Record<string, number>, margin = 0): boolean {32  return [...rates].every(([category, rate]) => rate <= (baseline[category] ?? 0) + margin);33}
33 linesUTF-8TypeScript
mutate(probe.prompt)
Each attack runs in many variations, since models that block one phrasing often miss another.
failures.push(...)
Each success becomes tomorrow's regression test.
passesGate
The build fails if any category gets worse than its baseline.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why test the whole product?
Tools and retrieved content create risks the bare model doesn't have, such as indirect injection.
How do you stop the suite from going stale?
Feed in incidents and new public attacks, and replace probes that no longer reveal anything.
How do you keep the probes safe?
Restrict access, log usage and never ship them in client code.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Costs
Thousands of attempts per release cost money. Run the full suite before release and a smoke subset per change.
Counterpoint
A clean suite means known attacks failed, not that the product is safe. Keep human red-teamers for new thinking.
AI Engineer · Automated Adversarial Red-Teaming Suites