/Learn
Deep dives · AI handbook · customisation

Prompting, RAG or fine-tuning?

LearnDeep diveTier 2 · MechanismsAnalyze20 minFine-tuning

You will be able to: explain prompting, RAG or fine-tuning? and answer its interview questions.

Prove it: PyTorch LoRA Linear Layer Implementation

AI engineeringDeep dives

Study with AI

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

Three levers fit a general model to your product: better prompts, retrieval at query time, and new weights. Prompts and retrieval change what the model sees; fine-tuning changes how it behaves.

Use each when
  • Prompting: format, tone, rules and few-shot examples
  • RAG: facts that change, citations and permissions
  • Fine-tuning: consistent style, narrow tasks, lower latency with a smaller model
  • Preference tuning: choosing between acceptable answers the way your users prefer
Common mistakes
  • Fine-tuning to teach facts that change weekly
  • RAG for behaviour the model should always show
  • Skipping evaluation because the demo looked good
  • Training on data you are not allowed to use

1.2Choosing for three real features

A concrete example first, so the mechanics have something to attach to.

Feature one, a refund-policy assistant, needs current facts with citations and permissions. That is retrieval: policies change and must be quoted. Fine-tuning would freeze yesterday’s policy into weights.

Feature two, ticket classification into 40 categories at 2 million tickets a day, needs speed and low cost more than broad reasoning. A small model fine-tuned with LoRA on 20,000 labelled tickets matches the large model’s accuracy on this task at a fraction of the cost.

Feature three, marketing copy in a strict brand voice, first gets a strong system prompt with examples. When reviewers still edit a third of outputs, the team collects pairs of preferred and rejected drafts and applies preference tuning (DPO) to an open model.

All three share one evaluation harness, so numbers back each decision, not impressions.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. When would you fine-tune instead of RAG?
  2. What is LoRA and why is it popular?
  3. How do you know fine-tuning helped?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1The ladder of customisation

The parts and how they connect.

1 · Promptinginstructions ·examples2 · RAGfresh facts ·citations3 · Fine-tuningSFT · LoRA4 · PreferenceDPO · RLHFfactshabitstaste
ServiceAI component
Boxes are components; arrows show calls and data flow.
Start at the left
Each step right costs more data, more engineering and more maintenance. Move right only when evaluation shows the previous step is not enough.
Supervised fine-tuning (SFT)
Train on input → ideal output pairs. Good for formats, classification, extraction and a consistent voice.
LoRA and QLoRA
Train small low-rank adapter matrices instead of all weights. QLoRA does this on a 4-bit quantised base model, so large models can be tuned on modest GPUs. Many adapters can share one base model in serving.
Preference tuning
DPO learns directly from pairs of preferred and rejected answers. RLHF trains a reward model and optimises against it; more complex, used by model providers.
Distillation
Generate high-quality outputs with a large model, then fine-tune a small model on them for a narrow task.

3.2Fine-tuning methods

NameMeaning
Full fine-tuneUpdate all weights. Highest flexibility, highest cost and risk of forgetting.
LoRAAdd trainable low-rank matrices to attention and MLP layers; typically well under 1% of parameters.
QLoRALoRA on a 4-bit base; fits large models on fewer GPUs.
DPOPreference pairs, no separate reward model.
Provider fine-tuningHosted APIs that fine-tune supported models on uploaded examples.

3.3A LoRA fine-tuning pipeline

One run, step by step.

Data teamDatasetTrainerEvaluatorServingcollect 20k labelled examplesdedupe, split train/val/testLoRA on base modeladapter checkpointscore vs baseline on test setaccuracy, cost, latencyPromote only if it beats prompting + RAGdeploy adapter on shared base
Solid arrows are calls; dashed arrows are responses.
  1. Hold out a test set that the model never sees, and compare against the cheapest baseline, not just the original prompt.
  2. Version the dataset, base model and adapter together.

3.4Code

Working code for the core of it.

LoRA in numbers: what you actually train (TypeScript)

lora-numbers-what.tsTypeScript
1// LoRA freezes W (dOut × dIn) and trains two thin matrices: W' = W + (alpha / r) · B·A,2// with B: dOut × r and A: r × dIn. Only B and A are saved, as a small adapter file.3const loraParams = (dIn: number, dOut: number, r: number) => r * (dIn + dOut);   // full: dIn × dOut4 5// Llama 3.1 8B: 32 layers, hidden size 4096, grouped-query attention (k and v project to 1024).6const layers = 32, hidden = 4096, kv = 1024, r = 16;7const perLayer =8  loraParams(hidden, hidden, r) +     // q_proj9  loraParams(hidden, kv, r) +         // k_proj10  loraParams(hidden, kv, r) +         // v_proj11  loraParams(hidden, hidden, r);      // o_proj12const trainable = layers * perLayer;  // 13,631,48813console.log(trainable.toLocaleString(), ((trainable / 8.03e9) * 100).toFixed(2) + "% of the weights");   // ≈ 0.17%
13 linesUTF-8TypeScript

Training example format

training-example-format.jsonJSON
1{ "messages": [2  { "role": "system", "content": "Classify the support ticket. Reply with one category id." },3  { "role": "user", "content": "I was charged twice for my annual plan" },4  { "role": "assistant", "content": "billing.duplicate_charge" } ] }
4 linesUTF-8JSON

3.5Tested code, explained

A working implementation, then what each part does and why.

How few weights LoRA trains

This is the whole reason LoRA is cheap, in one formula.

lora-params.tsTypeScript
1/** LoRA trains two thin matrices: B (dOut × r) and A (r × dIn). */2export const loraParams = (dIn: number, dOut: number, r: number) => r * (dIn + dOut);3 4/** Trainable weights when LoRA wraps the given projections in every layer. */5export function loraTrainable(layers: number, projections: [dIn: number, dOut: number][], r: number): number {6  return layers * projections.reduce((sum, [dIn, dOut]) => sum + loraParams(dIn, dOut, r), 0);7}
7 linesUTF-8TypeScript
r * (dIn + dOut)
Two thin matrices, rather than one of dIn by dOut. For 4096 by 4096 at rank 16, that's 131,072 weights instead of about 16.8 million.

Tested by npm run test:modules.

A LoRA layer from scratch

The layer behind PEFT, written out in full.

lora_linear.pyPython
1import math2 3import torch4from torch import nn5 6 7class LoRALinear(nn.Module):8    """Wraps a frozen nn.Linear with a trainable low-rank update: W + (alpha / r) * B @ A."""9 10    def __init__(self, base: nn.Linear, r: int = 16, alpha: float = 32.0, dropout: float = 0.05):11        super().__init__()12        self.base = base13        self.base.requires_grad_(False)14        self.lora_A = nn.Parameter(torch.empty(r, base.in_features))15        self.lora_B = nn.Parameter(torch.zeros(base.out_features, r))  # zero: starts as the base model16        nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))17        self.scale = alpha / r18        self.dropout = nn.Dropout(dropout)19        self.merged = False20 21    def forward(self, x: torch.Tensor) -> torch.Tensor:22        out = self.base(x)23        if self.merged:24            return out25        return out + (self.dropout(x) @ self.lora_A.T @ self.lora_B.T) * self.scale26 27    @torch.no_grad()28    def merge(self) -> None:29        if not self.merged:30            self.base.weight += self.scale * (self.lora_B @ self.lora_A)31            self.merged = True
31 linesUTF-8Python
requires_grad_(False)
The original weights stay frozen.
torch.zeros(...)
B starts at zero, so training begins from exactly the base model.
alpha / r
Scaling that keeps the learning rate stable when you change the rank.
def merge
Bakes the update into the weights, so inference has no extra cost.

Syntax-checked. The GPU libraries it needs don't run in our tests.

Merging an adapter

For serving, you bake the adapter in. For switching, you keep it separate.

merge-adapter.tsTypeScript
1type Matrix = number[][];2 3const mul = (a: Matrix, b: Matrix): Matrix => a.map((row) => b[0].map((_, j) => row.reduce((s, x, k) => s + x * b[k][j], 0)));4const combine = (w: Matrix, d: Matrix, sign: 1 | -1, scale: number): Matrix => w.map((row, i) => row.map((x, j) => x + sign * scale * d[i][j]));5 6/** W' = W + (alpha / r) * B @ A: bake the adapter into the weights. */7export const mergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), 1, alpha / a.length);8 9/** The exact inverse, up to floating-point error: keep the original weights if you must be exact. */10export const unmergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), -1, alpha / a.length);11 12/** Output with the adapter still separate: W x + (alpha / r) * B (A x). */13export function applyUnmerged(w: Matrix, a: Matrix, b: Matrix, alpha: number, x: number[]): number[] {14  const col = x.map((v) => [v]);15  const base = mul(w, col);16  const delta = mul(b, mul(a, col));17  return base.map((row, i) => row[0] + (alpha / a.length) * delta[i][0]);18}
18 linesUTF-8TypeScript
(alpha / r) * B @ A
The update LoRA learned, added to the weights.
unmergeAdapter
Subtracts it again, up to floating-point error. Keep the original weights if you need them exact.

Tested by npm run test:modules.

The DPO loss

It trains on pairs of answers, with no separate reward model.

dpo-loss.tsTypeScript
1export interface PairLogProbs {2  chosen: number;3  rejected: number;4  refChosen: number;5  refRejected: number;6}7 8/** DPO: -log sigmoid(beta * [(chosen - refChosen) - (rejected - refRejected)]). No reward model. */9export function dpoLoss(p: PairLogProbs, beta = 0.1): number {10  const logits = beta * (p.chosen - p.refChosen - (p.rejected - p.refRejected));11  return Math.log1p(Math.exp(-logits));12}
12 linesUTF-8TypeScript
p.chosen - p.refChosen
How much the model has moved towards the good answer, compared with the frozen reference.
Math.log1p(Math.exp(-logits))
A numerically stable form of minus log sigmoid.
beta
Controls how far the model may drift from the reference.

Tested by npm run test:modules.

3.6Data you need for each

Prompting

  • A versioned system prompt, 3–10 good examples, an evaluation set.

RAG

  • Clean, chunked documents with metadata and permissions; labelled questions for retrieval evaluation.

Fine-tuning

  • Thousands of high-quality input/output pairs representative of production; a held-out test set.

Preference tuning

  • Pairs of better and worse answers, ideally from real reviewer edits.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

When would you fine-tune instead of RAG?
When the problem is behaviour, format or cost on a narrow, high-volume task, not missing facts. Facts that change belong in retrieval.
What is LoRA and why is it popular?
It freezes the base model and trains small low-rank matrices, so training is cheap, adapters are small files, and many adapters can share one base in serving.
How do you know fine-tuning helped?
Compare against the best prompt + RAG baseline on a held-out test set, including cost and latency, and check for regressions on other skills.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Tools

Show detailsHide details
NeedOptionsWhen to pick
Training librariesHugging Face Transformers · PEFT · TRL · Axolotl · UnslothPEFT + TRL for LoRA, SFT and DPO; Unsloth or Axolotl for fast recipes.
Hosted fine-tuningFine-tuning APIs from major providers and cloud platformsWhen you want no training infrastructure and the base model is supported.
Serving adaptersvLLM multi-LoRA · TGIServe many adapters on one base model.
Experiment trackingWeights & Biases · MLflowVersion data, configs, metrics and checkpoints.

5.2How to judge the result

Show detailsHide details
MetricWhy it matters
Task accuracy / F1 on held-out testThe main decision metric.
Regression on general behaviourFine-tuning can degrade other skills; test them.
Cost and latency per requestOften the reason to fine-tune a small model.
Human preference win rateFor style and tone tasks.

5.3Risks

Show detailsHide details
CaseHow it is handled
Facts baked into weights go staleUse retrieval for anything that changes.
Training data leaks personal dataRedact before training; models can memorise.
Overfitting to a narrow datasetDiverse examples, held-out evaluation, early stopping.
Base model upgradeAdapters are tied to a base; plan to retrain.
AI Engineer · Prompting, RAG or fine-tuning?