Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
An unmerged adapter runs an extra small multiply per wrapped layer. Merging adds (α / r) · B · A into the weights once, so serving costs what the base costs. You lose the ability to swap adapters cheaply.
Bakes an adapter into the weights for faster serving, and shows when merging is the wrong call.
1.2Key concepts
One idea per card, in plain words.
W′ = W + (α / r) · B · A.
Subtracts the same delta. Floating-point error means keeping the original when you need exact weights.
PEFT's call that merges, then removes the wrapper layers.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Write merge and unmerge for one weight matrix, and a check that merged output equals unmerged output for any input.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Compute the delta
Multiply B by A, then scale by α / r.
- Add it to the weight
The shapes match by construction.
- Verify on sample inputs
Compare merged and unmerged outputs within a small tolerance.
- Save the merged model
Export the full weights, and keep the adapter separately for audit.
3.2Code
Working code for the core of it.
Merging an adapter
For serving, you bake the adapter in. For switching, you keep it separate.
1type Matrix = number[][];2 3const mul = (a: Matrix, b: Matrix): Matrix => a.map((row) => b[0].map((_, j) => row.reduce((s, x, k) => s + x * b[k][j], 0)));4const combine = (w: Matrix, d: Matrix, sign: 1 | -1, scale: number): Matrix => w.map((row, i) => row.map((x, j) => x + sign * scale * d[i][j]));5 6/** W' = W + (alpha / r) * B @ A: bake the adapter into the weights. */7export const mergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), 1, alpha / a.length);8 9/** The exact inverse, up to floating-point error: keep the original weights if you must be exact. */10export const unmergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), -1, alpha / a.length);11 12/** Output with the adapter still separate: W x + (alpha / r) * B (A x). */13export function applyUnmerged(w: Matrix, a: Matrix, b: Matrix, alpha: number, x: number[]): number[] {14 const col = x.map((v) => [v]);15 const base = mul(w, col);16 const delta = mul(b, mul(a, col));17 return base.map((row, i) => row[0] + (alpha / a.length) * delta[i][0]);18}(alpha / r) * B @ A- The update LoRA learned, added to the weights.
unmergeAdapter- Subtracts it again, up to floating-point error. Keep the original weights if you need them exact.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
When should you merge?
When should you not?
Can you merge into a 4-bit model?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Rounding
- FP16 merges accumulate small errors. Merge at FP32 when precision matters.
- Counterpoint
- Modern engines serve unmerged adapters with little overhead. Measure the latency gain before giving up flexibility.