/Learn
Modules · Evals and guardrails · Machine coding

Llama-Guard Jailbreak & Safety Classifier

PracticeMachine codingTier 2 · MechanismsApply30 minEvals and guardrails

You will be able to: build a moderation gate with per-product policy, safe failure modes and measurable false positives.

Source: Inan et al., 2023 (Llama Guard); Meta model cards

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A small classifier reads each message and replies 'safe', or 'unsafe' plus category codes. Your code decides what those codes mean for this product, and what happens if the classifier itself fails.

Why it matters

Moderates input and output with a dedicated classifier, and fails closed whenever actions are at stake.

1.2Key concepts

One idea per card, in plain words.

01Llama Guard

A safety model that labels text using a harm taxonomy such as S1–S14.

02Input and output rails

Check the user's message before the model, and the reply after it.

03Policy mapping

A medical product may allow health topics that a children's product blocks. The product, not the classifier, decides.

04Fail closed

If the classifier errors, block anything that takes an action.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Add a moderation gate in front of your assistant. Parse the classifier's output, apply a per-product allow list, and define behaviour when the classifier is down.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Can the assistant take actions?Yes, it books and cancels. That decides the failure mode.
What latency budget?About 50 ms on input. Stream output through a lighter check.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Classify the input

    Send the message in the model's expected prompt format.

  2. Parse strictly

    First line 'safe' or 'unsafe'; second line holds category codes. Treat anything else as an error.

  3. Apply product policy

    Remove categories this product allows, then block on what remains.

  4. Choose failure modes

    Fail closed for actions. A read-only FAQ may fail open with logging.

  5. Measure both errors

    Track missed harms and over-blocking, per category.

3.2Code

Working code for the core of it.

A safety gate before the model

It runs before the expensive model, and has to decide what happens if it fails.

moderation-gate.tsTypeScript
1export interface Verdict {2  allow: boolean;3  categories: string[];4  reason: "safe" | "unsafe" | "blocked-category" | "classifier-error";5}6 7/** Llama Guard replies "safe" or "unsafe" followed by category codes such as "S1,S10". */8export function parseGuardOutput(text: string): { safe: boolean; categories: string[] } {9  const [first, second = ""] = text.trim().split("\n");10  const safe = first.trim().toLowerCase() === "safe";11  return { safe, categories: safe ? [] : second.split(",").map((c) => c.trim()).filter(Boolean) };12}13 14/** Runs before the main model. allowed lists categories this product accepts despite the flag. */15export async function gate(input: string, classify: (text: string) => Promise<string>, allowed: string[] = [], failOpen = false): Promise<Verdict> {16  try {17    const { safe, categories } = parseGuardOutput(await classify(input));18    if (safe) return { allow: true, categories: [], reason: "safe" };19    const blocked = categories.filter((c) => !allowed.includes(c));20    if (categories.length > 0 && blocked.length === 0) return { allow: true, categories, reason: "safe" };21    return { allow: false, categories: blocked, reason: blocked.length ? "blocked-category" : "unsafe" };22  } catch {23    // Actions fail closed. A read-only product can choose to fail open.24    return { allow: failOpen, categories: [], reason: "classifier-error" };25  }26}
26 linesUTF-8TypeScript
parseGuardOutput
Llama Guard replies safe, or unsafe plus category codes.
allowed
A medical product may accept a category that a children's product blocks.
failOpen = false
If the classifier errors, products that take actions should refuse.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Does this stop prompt injection?
Partly. Injection classifiers catch known patterns, not all. Limit what the model can do, and confirm sensitive actions.
Why a separate model?
It runs fast, updates independently, and doesn't depend on the main model's judgement.
How do you handle false positives?
Review blocked samples weekly, adjust policy, and offer users a way forward.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Output streaming
Check complete sentences as they stream, or buffer sensitive flows.
Counterpoint
Classifiers miss new attacks and block innocent text. Treat them as one layer, never the whole defence.
AI Engineer · Llama-Guard Jailbreak & Safety Classifier