Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Full fine-tuning updates billions of weights. LoRA freezes them and learns a small correction for each chosen matrix: W′ = W + (α / r) · B · A. B and A stay thin, so you train a fraction of a percent of the parameters and save a small adapter file.
Trains two thin matrices instead of every weight, so fine-tuning fits one GPU and merges with no added inference latency.
1.2Key concepts
One idea per card, in plain words.
Approximates a large change with the product of two thin matrices, B (d_out × r) and A (r × d_in).
Sets the adapter's capacity. Common values run from 8 to 32.
Scales the update through α / r, so changing r doesn't force a new learning rate.
B starts at zero, so training begins from the unchanged base model.
Adds the update into W for serving. The merged model runs as fast as the base.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Your team wants a domain-specific tone from an 8B model, on one 24 GB GPU. Explain how LoRA makes that possible, how many weights you train, and which layers you target.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Style or facts? | Style and format suit fine-tuning. Changing facts suit retrieval. |
| How much data? | A few hundred to a few thousand good examples. |
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Freeze the base
Set requires_grad to false for every base weight. Nothing else changes in the forward pass.
- Wrap target layers
Add A and B around chosen linear layers. Attention projections are the common choice.
- Train the adapter
Only A and B receive gradients, so memory use and checkpoint size shrink sharply.
- Save only the adapter
The file holds A and B, at tens of megabytes, and loads onto the base later.
- Merge or keep separate
Merge for one fixed adapter. Keep separate to swap many adapters on one base.
3.2Trainable weights, Llama 3.1 8B
32 layers, wrapping the q, k, v and output projections. Full-rank training would touch all 8 billion weights.
| Rank | Trainable | Share of model |
|---|---|---|
| 8 | 6.8 million | 0.08% |
| 16 | 13.6 million | 0.17% |
| 32 | 27.3 million | 0.34% |
3.3Code
Working code for the core of it.
How few weights LoRA trains
This is the whole reason LoRA is cheap, in one formula.
1/** LoRA trains two thin matrices: B (dOut × r) and A (r × dIn). */2export const loraParams = (dIn: number, dOut: number, r: number) => r * (dIn + dOut);3 4/** Trainable weights when LoRA wraps the given projections in every layer. */5export function loraTrainable(layers: number, projections: [dIn: number, dOut: number][], r: number): number {6 return layers * projections.reduce((sum, [dIn, dOut]) => sum + loraParams(dIn, dOut, r), 0);7}r * (dIn + dOut)- Two thin matrices, rather than one of dIn by dOut. For 4096 by 4096 at rank 16, that's 131,072 weights instead of about 16.8 million.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why does a low-rank update work at all?
Does LoRA slow inference?
How do you pick r?
LoRA or RAG?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Target modules
- Wrapping every linear layer rarely repays the extra weights. Start with attention projections.
- Serving many adapters
- vLLM and others load adapters per request, so one GPU can serve many customers.
- Counterpoint
- Prompting and retrieval solve most needs at far lower cost. Fine-tune only once evals show they fall short.