Academy

Why LoRA works: the intrinsic-dimensionality story

If a large model has billions of parameters, how can you tune it by training only a few small matrices? The answer is a subtle idea: the change you need has a small intrinsic dimension.

The Felesh team 4 min

LoRA is remarkably effective in practice: it tunes a giant model by training only a few small matrices while the main body remains frozen. But why does this work? How can you meaningfully alter the behaviour of a billion-parameter model without touching the vast majority of its weights? The answer lies in a single, subtle insight.

Two kinds of knowledge, two kinds of need

To understand LoRA, we must distinguish between two phases of training. Pre-training is expansive: the model learns broad general knowledge from scratch — language, facts, and patterns. This requires full capacity and uses every one of those billions of parameters. Fine-tuning, however, is narrow: adapting an already-capable model for a specific task. That adaptation is fundamentally less complex.

Pre-training moves every weight; fine-tuning moves a fewTwo panels. Pre-training starts from scratch, and every weight in its grid changes across billions of parameters. Fine-tuning starts from a capable model, where most weights stay put; two of them change to tag the ticket “Charged twice for March.” as refund.pre-trainingfrom scratchfine-tuningfrom a capable modelall weights trainbillions of parameters“Charged twice for March.”→ tag: refund
Pre-training moves every weight to build general knowledge; fine-tuning starts from a capable model and moves only what one task needs.

The low-intrinsic-dimension hypothesis

LoRA is built on a specific premise: although a model’s original weights are high-dimensional, the change required for fine-tuning has a low intrinsic dimension. The updates necessary for fine-tuning fit into a small space, even when the underlying model is enormous. This empirical observation forms the foundation of the method. You do not need to alter the entire model, because the useful change is inherently narrow.

The simple math: ΔW = BA

LoRA implements this idea directly. Instead of training a large, full-rank change matrix (ΔW), it expresses the update as the product of two small matrices: ΔW = B × A, where A and B have a low rank. Instead of training millions of values, you train only a few thousand. The final output is the original, frozen weight plus this trained low-rank change. The reduction in trainable parameters is dramatic — often hundreds of times.

ΔW is the product of two thin matricesFor one 4096 by 4096 layer, a full update ΔW would hold 16,777,216 values. LoRA trains A, 8 by 4096 with 32,768 values, and B, 4096 by 8 with 32,768 values. Their product B times A has the full shape, while the trained part is 65,536 values, one 256th of the full update.ΔW = B × AA · 8 × 409632,768 valuesB · 4096 × 832,768 values65,536 trained1/256 of the full updateΔW: 16,777,216 values
LoRA trains two thin matrices holding 65 thousand values in total; their product fills the same 4096 by 4096 grid that a full update would need.

The initialisation trick

There is an elegant detail in the implementation. At the start of training, matrix A is initialised randomly and matrix B is initialised with zeros. As a result, the product B × A is exactly zero at the start — meaning the model behaves identically to the base model on the first step. Training begins from the exact state of the original model and gradually diverges. This initialisation trick prevents a sudden, destructive shift in behaviour at the onset of training.

Zero B makes the first step the base modelA is random, 2 by 4, and B is zero, 4 by 2. Their product ΔW is zero, so the output equals the base model’s output. After one update B holds values off zero, ΔW is no longer zero, and the output starts to drift from the base model.A · randomB · zeroΔW = B × A0.31 −0.52 0.08 0.77−0.14 0.26 −0.63 0.400 00 00 00 00.2 −0.10.3 0.1−0.2 0.40.1 −0.3every value 0non-zero×=step 0: B = 0, so ΔW = 0step 1: B moves off zerooutput = W·x: the base model, exactlyoutput = W·x + ΔW·x: drift starts
B starts at zero, so ΔW = B × A is zero and the first step returns exactly the base model's output; training moves B off zero from there.

Why this matters

Understanding this mechanism is not just a theoretical curiosity; it shapes practical engineering decisions. Because the necessary change has a low intrinsic dimension, setting a very high rank is usually unnecessary and only increases the risk of overfitting. Furthermore, because the main body of weights remains untouched, the model’s general knowledge is preserved, significantly reducing the risk of catastrophic forgetting. LoRA, in effect, ensures that what is theoretically small remains structurally small.

Trainable values grow with the rankA table of four ranks for one 4096 by 4096 layer. Rank 8 trains 65,536 values, 0.4 percent of the full update. Rank 64 trains 524,288, 3.1 percent. Rank 512 trains 4,194,304, 25 percent. Rank 2,048 trains 16,777,216, which is the full update. The frozen weights keep their 16,777,216 values throughout.ranktrained valuesshare of the full update865,5360.4%64524,2883.1%5124,194,30425%2,04816,777,216100%trained = 2 × 4096 × rank · W stays frozen
Trainable values grow in step with the rank, so a rank of 2,048 already trains as many values as the full 4096 by 4096 update.

Putting it together

LoRA stems from a simple insight: adapting a model for a specific task does not require a massive structural change, because the necessary update inherently fits within a narrow space. By expressing this change as the product of two small matrices, you can tune enormous models at a fraction of the traditional computational cost — without degrading the model’s core knowledge. That’s the beauty of a right idea in the right place.