Basics
What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.
1.1Definition
The one-paragraph summary, then where its responsibility starts and stops.
Three levers fit a general model to your product: better prompts, retrieval at query time, and new weights. Prompts and retrieval change what the model sees; fine-tuning changes how it behaves.
- Prompting: format, tone, rules and few-shot examples
- RAG: facts that change, citations and permissions
- Fine-tuning: consistent style, narrow tasks, lower latency with a smaller model
- Preference tuning: choosing between acceptable answers the way your users prefer
- Fine-tuning to teach facts that change weekly
- RAG for behaviour the model should always show
- Skipping evaluation because the demo looked good
- Training on data you are not allowed to use
1.2Choosing for three real features
A concrete example first, so the mechanics have something to attach to.
Feature one, a refund-policy assistant, needs current facts with citations and permissions. That is retrieval: policies change and must be quoted. Fine-tuning would freeze yesterday’s policy into weights.
Feature two, ticket classification into 40 categories at 2 million tickets a day, needs speed and low cost more than broad reasoning. A small model fine-tuned with LoRA on 20,000 labelled tickets matches the large model’s accuracy on this task at a fraction of the cost.
Feature three, marketing copy in a strict brand voice, first gets a strong system prompt with examples. When reviewers still edit a third of outputs, the team collects pairs of preferred and rejected drafts and applies preference tuning (DPO) to an open model.
All three share one evaluation harness, so numbers back each decision, not impressions.
Question
Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.
2.1Target questions
Keep them in mind while you read the detail.
- When would you fine-tune instead of RAG?
- What is LoRA and why is it popular?
- How do you know fine-tuning helped?
Model answers are in part 4, after the detail that supports them.
2.2Interview rounds
System design rounds where this component is part of a strong answer.
Detailed answer
How it actually works: the components, the states it moves through, a full run, and the code.
3.1The ladder of customisation
The parts and how they connect.
- Start at the left
- Each step right costs more data, more engineering and more maintenance. Move right only when evaluation shows the previous step is not enough.
- Supervised fine-tuning (SFT)
- Train on input → ideal output pairs. Good for formats, classification, extraction and a consistent voice.
- LoRA and QLoRA
- Train small low-rank adapter matrices instead of all weights. QLoRA does this on a 4-bit quantised base model, so large models can be tuned on modest GPUs. Many adapters can share one base model in serving.
- Preference tuning
- DPO learns directly from pairs of preferred and rejected answers. RLHF trains a reward model and optimises against it; more complex, used by model providers.
- Distillation
- Generate high-quality outputs with a large model, then fine-tune a small model on them for a narrow task.
3.2Fine-tuning methods
| Name | Meaning |
|---|---|
| Full fine-tune | Update all weights. Highest flexibility, highest cost and risk of forgetting. |
| LoRA | Add trainable low-rank matrices to attention and MLP layers; typically well under 1% of parameters. |
| QLoRA | LoRA on a 4-bit base; fits large models on fewer GPUs. |
| DPO | Preference pairs, no separate reward model. |
| Provider fine-tuning | Hosted APIs that fine-tune supported models on uploaded examples. |
3.3A LoRA fine-tuning pipeline
One run, step by step.
- Hold out a test set that the model never sees, and compare against the cheapest baseline, not just the original prompt.
- Version the dataset, base model and adapter together.
3.4Code
Working code for the core of it.
LoRA in numbers: what you actually train (TypeScript)
1// LoRA freezes W (dOut × dIn) and trains two thin matrices: W' = W + (alpha / r) · B·A,2// with B: dOut × r and A: r × dIn. Only B and A are saved, as a small adapter file.3const loraParams = (dIn: number, dOut: number, r: number) => r * (dIn + dOut); // full: dIn × dOut4 5// Llama 3.1 8B: 32 layers, hidden size 4096, grouped-query attention (k and v project to 1024).6const layers = 32, hidden = 4096, kv = 1024, r = 16;7const perLayer =8 loraParams(hidden, hidden, r) + // q_proj9 loraParams(hidden, kv, r) + // k_proj10 loraParams(hidden, kv, r) + // v_proj11 loraParams(hidden, hidden, r); // o_proj12const trainable = layers * perLayer; // 13,631,48813console.log(trainable.toLocaleString(), ((trainable / 8.03e9) * 100).toFixed(2) + "% of the weights"); // ≈ 0.17%Training example format
1{ "messages": [2 { "role": "system", "content": "Classify the support ticket. Reply with one category id." },3 { "role": "user", "content": "I was charged twice for my annual plan" },4 { "role": "assistant", "content": "billing.duplicate_charge" } ] }3.5Tested code, explained
A working implementation, then what each part does and why.
How few weights LoRA trains
This is the whole reason LoRA is cheap, in one formula.
1/** LoRA trains two thin matrices: B (dOut × r) and A (r × dIn). */2export const loraParams = (dIn: number, dOut: number, r: number) => r * (dIn + dOut);3 4/** Trainable weights when LoRA wraps the given projections in every layer. */5export function loraTrainable(layers: number, projections: [dIn: number, dOut: number][], r: number): number {6 return layers * projections.reduce((sum, [dIn, dOut]) => sum + loraParams(dIn, dOut, r), 0);7}r * (dIn + dOut)- Two thin matrices, rather than one of dIn by dOut. For 4096 by 4096 at rank 16, that's 131,072 weights instead of about 16.8 million.
Tested by npm run test:modules.
A LoRA layer from scratch
The layer behind PEFT, written out in full.
1import math2 3import torch4from torch import nn5 6 7class LoRALinear(nn.Module):8 """Wraps a frozen nn.Linear with a trainable low-rank update: W + (alpha / r) * B @ A."""9 10 def __init__(self, base: nn.Linear, r: int = 16, alpha: float = 32.0, dropout: float = 0.05):11 super().__init__()12 self.base = base13 self.base.requires_grad_(False)14 self.lora_A = nn.Parameter(torch.empty(r, base.in_features))15 self.lora_B = nn.Parameter(torch.zeros(base.out_features, r)) # zero: starts as the base model16 nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))17 self.scale = alpha / r18 self.dropout = nn.Dropout(dropout)19 self.merged = False20 21 def forward(self, x: torch.Tensor) -> torch.Tensor:22 out = self.base(x)23 if self.merged:24 return out25 return out + (self.dropout(x) @ self.lora_A.T @ self.lora_B.T) * self.scale26 27 @torch.no_grad()28 def merge(self) -> None:29 if not self.merged:30 self.base.weight += self.scale * (self.lora_B @ self.lora_A)31 self.merged = Truerequires_grad_(False)- The original weights stay frozen.
torch.zeros(...)- B starts at zero, so training begins from exactly the base model.
alpha / r- Scaling that keeps the learning rate stable when you change the rank.
def merge- Bakes the update into the weights, so inference has no extra cost.
Syntax-checked. The GPU libraries it needs don't run in our tests.
Merging an adapter
For serving, you bake the adapter in. For switching, you keep it separate.
1type Matrix = number[][];2 3const mul = (a: Matrix, b: Matrix): Matrix => a.map((row) => b[0].map((_, j) => row.reduce((s, x, k) => s + x * b[k][j], 0)));4const combine = (w: Matrix, d: Matrix, sign: 1 | -1, scale: number): Matrix => w.map((row, i) => row.map((x, j) => x + sign * scale * d[i][j]));5 6/** W' = W + (alpha / r) * B @ A: bake the adapter into the weights. */7export const mergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), 1, alpha / a.length);8 9/** The exact inverse, up to floating-point error: keep the original weights if you must be exact. */10export const unmergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), -1, alpha / a.length);11 12/** Output with the adapter still separate: W x + (alpha / r) * B (A x). */13export function applyUnmerged(w: Matrix, a: Matrix, b: Matrix, alpha: number, x: number[]): number[] {14 const col = x.map((v) => [v]);15 const base = mul(w, col);16 const delta = mul(b, mul(a, col));17 return base.map((row, i) => row[0] + (alpha / a.length) * delta[i][0]);18}(alpha / r) * B @ A- The update LoRA learned, added to the weights.
unmergeAdapter- Subtracts it again, up to floating-point error. Keep the original weights if you need them exact.
Tested by npm run test:modules.
The DPO loss
It trains on pairs of answers, with no separate reward model.
1export interface PairLogProbs {2 chosen: number;3 rejected: number;4 refChosen: number;5 refRejected: number;6}7 8/** DPO: -log sigmoid(beta * [(chosen - refChosen) - (rejected - refRejected)]). No reward model. */9export function dpoLoss(p: PairLogProbs, beta = 0.1): number {10 const logits = beta * (p.chosen - p.refChosen - (p.rejected - p.refRejected));11 return Math.log1p(Math.exp(-logits));12}p.chosen - p.refChosen- How much the model has moved towards the good answer, compared with the frozen reference.
Math.log1p(Math.exp(-logits))- A numerically stable form of minus log sigmoid.
beta- Controls how far the model may drift from the reference.
Tested by npm run test:modules.
3.6Data you need for each
Prompting
- A versioned system prompt, 3–10 good examples, an evaluation set.
RAG
- Clean, chunked documents with metadata and permissions; labelled questions for retrieval evaluation.
Fine-tuning
- Thousands of high-quality input/output pairs representative of production; a held-out test set.
Preference tuning
- Pairs of better and worse answers, ideally from real reviewer edits.
Q&A
The questions from part 2, now with model answers. Try each one out loud before opening it.
4.1Interview questions
When would you fine-tune instead of RAG?
What is LoRA and why is it popular?
How do you know fine-tuning helped?
Extra depth
Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.
5.1Tools
Show detailsHide details
| Need | Options | When to pick |
|---|---|---|
| Training libraries | Hugging Face Transformers · PEFT · TRL · Axolotl · Unsloth | PEFT + TRL for LoRA, SFT and DPO; Unsloth or Axolotl for fast recipes. |
| Hosted fine-tuning | Fine-tuning APIs from major providers and cloud platforms | When you want no training infrastructure and the base model is supported. |
| Serving adapters | vLLM multi-LoRA · TGI | Serve many adapters on one base model. |
| Experiment tracking | Weights & Biases · MLflow | Version data, configs, metrics and checkpoints. |
5.2How to judge the result
Show detailsHide details
| Metric | Why it matters |
|---|---|
| Task accuracy / F1 on held-out test | The main decision metric. |
| Regression on general behaviour | Fine-tuning can degrade other skills; test them. |
| Cost and latency per request | Often the reason to fine-tune a small model. |
| Human preference win rate | For style and tone tasks. |
5.3Risks
Show detailsHide details
| Case | How it is handled |
|---|---|
| Facts baked into weights go stale | Use retrieval for anything that changes. |
| Training data leaks personal data | Redact before training; models can memorise. |
| Overfitting to a narrow dataset | Diverse examples, held-out evaluation, early stopping. |
| Base model upgrade | Adapters are tied to a base; plan to retrain. |