/Learn
Modules · Fine-tuning · Coding pattern

Merging LoRA Adapters into Base Model Weights

PracticeCoding patternTier 2 · MechanismsCreate10 minFine-tuning

You will be able to: merge and unmerge an adapter, and prove the merged output matches.

Source: Hugging Face PEFT documentation (merge_and_unload)

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

An unmerged adapter runs an extra small multiply per wrapped layer. Merging adds (α / r) · B · A into the weights once, so serving costs what the base costs. You lose the ability to swap adapters cheaply.

Why it matters

Bakes an adapter into the weights for faster serving, and shows when merging is the wrong call.

1.2Key concepts

One idea per card, in plain words.

01Merge

W′ = W + (α / r) · B · A.

02Unmerge

Subtracts the same delta. Floating-point error means keeping the original when you need exact weights.

03merge_and_unload

PEFT's call that merges, then removes the wrapper layers.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Write merge and unmerge for one weight matrix, and a check that merged output equals unmerged output for any input.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Compute the delta

    Multiply B by A, then scale by α / r.

  2. Add it to the weight

    The shapes match by construction.

  3. Verify on sample inputs

    Compare merged and unmerged outputs within a small tolerance.

  4. Save the merged model

    Export the full weights, and keep the adapter separately for audit.

3.2Code

Working code for the core of it.

Merging an adapter

For serving, you bake the adapter in. For switching, you keep it separate.

merge-adapter.tsTypeScript
1type Matrix = number[][];2 3const mul = (a: Matrix, b: Matrix): Matrix => a.map((row) => b[0].map((_, j) => row.reduce((s, x, k) => s + x * b[k][j], 0)));4const combine = (w: Matrix, d: Matrix, sign: 1 | -1, scale: number): Matrix => w.map((row, i) => row.map((x, j) => x + sign * scale * d[i][j]));5 6/** W' = W + (alpha / r) * B @ A: bake the adapter into the weights. */7export const mergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), 1, alpha / a.length);8 9/** The exact inverse, up to floating-point error: keep the original weights if you must be exact. */10export const unmergeAdapter = (w: Matrix, a: Matrix, b: Matrix, alpha: number): Matrix => combine(w, mul(b, a), -1, alpha / a.length);11 12/** Output with the adapter still separate: W x + (alpha / r) * B (A x). */13export function applyUnmerged(w: Matrix, a: Matrix, b: Matrix, alpha: number, x: number[]): number[] {14  const col = x.map((v) => [v]);15  const base = mul(w, col);16  const delta = mul(b, mul(a, col));17  return base.map((row, i) => row[0] + (alpha / a.length) * delta[i][0]);18}
18 linesUTF-8TypeScript
(alpha / r) * B @ A
The update LoRA learned, added to the weights.
unmergeAdapter
Subtracts it again, up to floating-point error. Keep the original weights if you need them exact.

Tested by npm run test:modules.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

When should you merge?
For one fixed adapter in production, or before export to a runtime without adapter support.
When should you not?
When one base serves many adapters; loading per request saves GPUs.
Can you merge into a 4-bit model?
Merge at FP16 or higher, then quantise the result.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Rounding
FP16 merges accumulate small errors. Merge at FP32 when precision matters.
Counterpoint
Modern engines serve unmerged adapters with little overhead. Measure the latency gain before giving up flexibility.
AI Engineer · Merging LoRA Adapters into Base Model Weights