Academy

The LoRA family: QLoRA, DoRA, and LoRA+ — which, and when?

Since LoRA was introduced, several improved variants have appeared, each targeting one particular problem. Knowing them helps you pick the right one for each job.

The Felesh team 3 min

LoRA is the baseline, but it is not the only option. Since its introduction, several variants have emerged to target specific constraints: memory, quality, or speed. Understanding these trade-offs helps you choose the right approach for a given workload.

QLoRA: when memory is tight

QLoRA is designed to save memory. It keeps the base model compressed at 4-bit precision (using the NF4 format), restoring it to higher precision only during computation. The impact is significant: a 7-billion-parameter model requires roughly 6 gigabytes of memory instead of about sixteen. Crucially, quality remains largely untouched, performing very closely to standard LoRA. If GPU VRAM is limited or you need to fine-tune a larger model on constrained hardware, QLoRA is the default choice.

One block at a time widens for its multiplyThe base model is a row of blocks, each stored at 4-bit. The blocks widen to 16-bit one after another, each only for its own multiply, then drop back to 4-bit. At rest every block is at 4-bit.one block at a time widens for its multiply16-bit16-bit16-bit16-bit4-bit4-bit4-bit4-bit4-bit4-bit4-bit4-bit16-bit only for the block being multiplied, then back to 4-bitblocks in order, layer by layer
QLoRA keeps every block of the base model at 4-bit and widens only the one being multiplied, so the model never sits in memory at 16-bit.

DoRA: when quality comes first

DoRA addresses a different problem: closing the quality gap between LoRA and full fine-tuning. It decomposes each weight into two components: magnitude and direction. DoRA manages these separately — applying LoRA to the direction while learning the magnitude independently. This separation improves quality by a few percent, particularly on complex tasks like reasoning and code generation, with under 5 percent overhead on the parameter count. If quality is critical and standard LoRA falls short, DoRA is the logical next step.

A weight column splits into a length and a directionThe weight column w = (3, 4) splits into a magnitude m = 5, trained on its own, and a unit direction (0.6, 0.8) where LoRA acts. After LoRA the direction reads (0.8, 0.6) while the magnitude is still trained separately. The whole change costs under 5 percent more parameters.w = (3, 4): one weight columnmagnitude m = 5trained on its owndirection (0.6, 0.8)direction (0.8, 0.6)LoRA moves this partunder 5% more parameters, a few points of quality back
DoRA splits each weight column into a length and a direction, so LoRA moves the direction while the length is learned on its own.

LoRA+: when speed matters

LoRA+ is a subtle but effective structural adjustment. In standard LoRA, matrix B initialises at zero and must travel further to reach its optimal value. LoRA+ compensates by assigning B a larger learning rate than A. The result is faster convergence — reaching target quality roughly 1.5 to 2 times faster. If the training budget is strict or rapid iteration is necessary, LoRA+ saves time with only a minor configuration change.

B reaches the target firstTwo training runs start with B at zero, and each step moves B by its learning rate. After four steps LoRA+ has reached the target; standard LoRA is two-thirds of the way and needs six steps for the same quality, 1.5 times as many.B covers its distance from zerotarget qualitystandard LoRA: one rate for A and BLoRA+: B gets the larger rateB starts at zeroat step 4, LoRA+ is on target; standard LoRA is at two-thirds
Giving B a larger learning rate than A lets B cover its distance from zero sooner, reaching the same quality roughly 1.5 to 2 times faster.

A simple decision tree

The choice reduces to a few operational questions:

  • Is the primary constraint memory? → QLoRA.
  • Is quality critical and standard LoRA shows a gap? → DoRA.
  • Is training speed the highest priority? → LoRA+.
  • None of the above? → standard LoRA, which remains highly effective for most workloads.

Putting it together

These variants are not rivals; they are specialised tools for specific constraints. They are also not mutually exclusive — their underlying mechanisms can be combined. But before adding complexity, start with the baseline: try standard LoRA, and only when you hit the limit of a specific constraint, reach for the variant that targets exactly that constraint.