/Learn
Modules · Fine-tuning · Coding pattern

Direct Preference Optimization (DPO) Loss Function

PracticeCoding patternTier 2 · MechanismsApply15 minFine-tuning

You will be able to: implement the DPO loss, and explain why it needs a frozen reference model.

Source: Rafailov et al., 2023 (DPO)

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

RLHF-PPO trains a reward model, then a policy against it, which proves fragile. DPO skips both: given a prompt and a preferred and rejected answer, it nudges the model to favour the preferred one relative to a frozen copy of itself.

Why it matters

Trains on preference pairs directly, with no separate reward model or PPO loop.

1.2Key concepts

One idea per card, in plain words.

01Preference pair

A prompt with a chosen and a rejected reply.

02Reference model

A frozen copy that anchors training, so the model can't drift far.

03Log-ratio

log π(y) − log π_ref(y): how much more the model likes a reply than its reference does.

04Beta

Scales the loss. Trainers often use about 0.1.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

You have 50,000 preference pairs from reviewers. Write the loss for one pair, given log-probabilities from the model and the frozen reference.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Score both replies

    Sum token log-probabilities for chosen and rejected under both models.

  2. Take the log-ratios

    Subtract reference from model for each reply.

  3. Compare them

    logits = β × (chosen ratio − rejected ratio).

  4. Apply logistic loss

    −log σ(logits). With no preference it equals ln 2, and it falls as the chosen ratio pulls ahead.

3.2Code

Working code for the core of it.

The DPO loss

It trains on pairs of answers, with no separate reward model.

dpo-loss.tsTypeScript
1export interface PairLogProbs {2  chosen: number;3  rejected: number;4  refChosen: number;5  refRejected: number;6}7 8/** DPO: -log sigmoid(beta * [(chosen - refChosen) - (rejected - refRejected)]). No reward model. */9export function dpoLoss(p: PairLogProbs, beta = 0.1): number {10  const logits = beta * (p.chosen - p.refChosen - (p.rejected - p.refRejected));11  return Math.log1p(Math.exp(-logits));12}
12 linesUTF-8TypeScript
p.chosen - p.refChosen
How much the model has moved towards the good answer, compared with the frozen reference.
Math.log1p(Math.exp(-logits))
A numerically stable form of minus log sigmoid.
beta
Controls how far the model may drift from the reference.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why keep a reference model?
It stops the model drifting: raising the preferred reply is only rewarded relative to where training began.
DPO or PPO?
DPO trains stably with ordinary supervised tooling. PPO keeps some edge if you have strong reward signals and online sampling.
What does beta do?
Lower values allow more change from the reference. Tune it with your eval suite.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Sum over tokens
Compute log-probabilities for reply tokens only, not the prompt.
Counterpoint
Preference data alone can make models over-verbose. Check your evals for length drift.
AI Engineer · Direct Preference Optimization (DPO) Loss Function