/Learn
Modules · Fine-tuning · Machine coding

PyTorch LoRA Linear Layer Implementation

PracticeMachine codingTier 2 · MechanismsCreate35 minFine-tuning

You will be able to: implement a LoRA linear layer from scratch, including merge, and explain each initialisation.

Source: Hu et al., 2021; Hugging Face PEFT internals

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

A LoRA layer wraps an existing linear layer, freezes it, and adds a small trainable path alongside. Output equals the base output plus the low-rank correction. Merging folds the correction into the base weight.

Why it matters

Writes a drop-in replacement for nn.Linear with trainable low-rank adapters and dropout.

1.2Key concepts

One idea per card, in plain words.

01Wrapping

Keeps the original layer untouched, so weights load unchanged.

02Kaiming init for A

Gives A random directions to learn along.

03Zero init for B

Makes the correction start at exactly zero.

04LoRA dropout

Applies dropout to the adapter path only, which regularises a small adapter.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Implement LoRALinear(base, r, alpha, dropout). The base stays frozen, output starts identical to the base, and merge() folds the update into the weight.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Bias?Keep the base bias frozen. The adapter adds none.
Which framework?PyTorch. The interface must match nn.Linear.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Freeze the base

    requires_grad_(False) covers the weight and any bias.

  2. Create A and B

    A has shape (r, in_features); B has shape (out_features, r). Their product matches the weight's shape.

  3. Initialise

    Kaiming for A and zeros for B, so the first forward pass equals the base.

  4. Forward

    Base output plus (dropout(x) @ Aᵀ @ Bᵀ) × α / r. Matrix order follows the batch-first input.

  5. Merge

    weight += (α / r) · B @ A, then skip the adapter path. Guard against merging twice.

3.2Code

Working code for the core of it.

A LoRA layer from scratch

The layer behind PEFT, written out in full.

lora_linear.pyPython
1import math2 3import torch4from torch import nn5 6 7class LoRALinear(nn.Module):8    """Wraps a frozen nn.Linear with a trainable low-rank update: W + (alpha / r) * B @ A."""9 10    def __init__(self, base: nn.Linear, r: int = 16, alpha: float = 32.0, dropout: float = 0.05):11        super().__init__()12        self.base = base13        self.base.requires_grad_(False)14        self.lora_A = nn.Parameter(torch.empty(r, base.in_features))15        self.lora_B = nn.Parameter(torch.zeros(base.out_features, r))  # zero: starts as the base model16        nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))17        self.scale = alpha / r18        self.dropout = nn.Dropout(dropout)19        self.merged = False20 21    def forward(self, x: torch.Tensor) -> torch.Tensor:22        out = self.base(x)23        if self.merged:24            return out25        return out + (self.dropout(x) @ self.lora_A.T @ self.lora_B.T) * self.scale26 27    @torch.no_grad()28    def merge(self) -> None:29        if not self.merged:30            self.base.weight += self.scale * (self.lora_B @ self.lora_A)31            self.merged = True
31 linesUTF-8Python
requires_grad_(False)
The original weights stay frozen.
torch.zeros(...)
B starts at zero, so training begins from exactly the base model.
alpha / r
Scaling that keeps the learning rate stable when you change the rank.
def merge
Bakes the update into the weights, so inference has no extra cost.

Syntax-checked. The GPU libraries it needs don't run in our tests.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why not zero both matrices?
Gradients for A depend on B and the reverse. With both at zero, neither moves.
What test proves it correct?
Before training, output equals the base. After random A and B, merged output equals unmerged output with dropout off.
What about model.eval()?
Dropout turns off, as usual. Merge only in eval, since merging assumes the full adapter path.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Replacing layers
Walk named_modules, match target names, and swap in the wrapper with setattr on the parent.
Counterpoint
PEFT already handles this, including quantised bases. Write your own to learn it or for an unusual layer, not by default.
AI Engineer · PyTorch LoRA Linear Layer Implementation