Academy

LoRA hyperparameters demystified: rank, alpha, and what to set

Fine-tuning with LoRA has a handful of key numbers, and once you know what they mean, choosing them is simple. This guide clears up rank, alpha, learning rate, and the rest.

The Felesh team 4 min

Fine-tuning with LoRA exposes a small surface area of hyperparameters, but getting them wrong often separates a useful model from gibberish. Once you understand the mechanical purpose of each, setting them stops being guesswork.

Rank: the size of what you learn

The rank (r) determines the dimensions of the low-rank matrices LoRA injects, which defines the model’s learning capacity. A larger rank absorbs more complex patterns but increases computational cost and the risk of overfitting. A practical baseline by task:

  • Style and formatting: r=4 to 8
  • Instruction following: r=8 to 16
  • Domain knowledge: r=16 to 32
  • Complex tasks: r=32 to 64

A sensible default is r=16. If the model underfits (training loss plateaus too high), increase the rank; if it overfits (validation loss diverges), decrease it.

Rank ranges by task on a doubling scaleFour task types with the rank ranges the post suggests, on a scale that doubles at each tick: style and formatting 4 to 8, instruction following 8 to 16, domain knowledge 16 to 32, complex tasks 32 to 64. The default r=16 is marked on the axis.style and formatting · r = 4 to 8instruction following · r = 8 to 16domain knowledge · r = 16 to 32complex tasks · r = 32 to 64rank r4816 (default)3264
The task sets the rank range on a doubling scale; start at r=16 and move one step only when the loss curves say so.

Alpha and the alpha-over-rank scale

Alpha (lora_alpha) has no standalone meaning; what matters is the ratio α/r, because LoRA’s contribution to the output scales by exactly this fraction. The standard practice is to set α = 2r (yielding α/r = 2). This ensures that if you adjust the rank later, LoRA’s relative contribution remains constant. If training grows unstable, α = r offers a more conservative scaling, and if you set α significantly higher, you must compensate by lowering the learning rate.

Doubling rank and alpha keeps the scaleRank 16 with alpha 32 gives a scale of 2.0, and a bar fills to 2.0 on an axis that marks alpha equal to r and alpha equal to 2r. Then the rank becomes 32 and alpha becomes 64: the numbers change, the scale stays at 2.0 and the bar does not move.ranklora_alphascale α/rr = 16r = 32α = 32α = 642.0α = rα = 2rsame scale: rank doubled, α doubled
Only the ratio α/r sets how much LoRA adds, so doubling both the rank and α keeps the scale at 2.

Learning rate

LoRA tolerates higher learning rates than full fine-tuning because it updates far fewer parameters. A standard functional range is 1e-4 to 3e-4. Starting at 2e-4 with a cosine decay schedule and roughly a three percent warmup is reliable. If your model outputs gibberish post-training, check this first; an excessive learning rate is the primary cause of catastrophic degradation.

A warmup, then a cosine decayLearning rate against training steps. It climbs over the first 3 percent of steps to a peak of 2e-4, then falls along a cosine curve to near zero at the last step. A dashed band marks the usual range from 1e-4 to 3e-4.usual range 1e-4 to 3e-4peak 2e-4warmup 3%cosine decaylast step
The learning rate climbs over a 3% warmup to 2e-4, then cosine decay brings it near zero by the last step.

Dropout

The dropout parameter mitigates overfitting. The optimal value scales inversely with dataset size:

  • Fewer than a thousand samples: 0.1
  • Between one and twenty thousand samples: 0.05
  • More than twenty thousand samples: 0.0 to 0.05

Sparse datasets carry a higher risk of memorisation, requiring more aggressive dropout to force the network to generalise.

Which layers: target_modules

Modern practice defaults to "all-linear", applying LoRA to every linear layer in the architecture. Older conventions that targeted only attention projections (q_proj and v_proj) have largely been deprecated in favour of the quality gains from comprehensive targeting. Attention layers handle routing, but MLP layers store factual associations; if a task requires net-new domain knowledge, both must be updated.

Which linear layers LoRA touchesOne transformer block has seven linear layers: four attention projections (q, k, v, o) that handle routing, and three MLP projections (gate, up, down) that store facts. Targeting only q_proj and v_proj lights two of seven. All-linear lights all seven.q_proj and v_proj only2 of 7 layersq_projk_projv_projo_projroutinggate_projup_projdown_projstores factsall-linear7 of 7 layersq_projk_projv_projo_projroutinggate_projup_projdown_projstores facts
Targeting only q_proj and v_proj leaves the MLP layers that store facts untouched; all-linear updates both.

Number of epochs

Epoch count should run inverse to dataset volume. Smaller datasets require more passes:

  • Fewer than a thousand samples: 3 to 5 epochs
  • One to ten thousand samples: 2 to 3 epochs
  • Ten to fifty thousand samples: 1 to 2 epochs
  • More than fifty thousand samples: 1 epoch

Over-iterating on large datasets reliably induces overfitting. A single epoch is frequently sufficient for production volume.

Putting it together: a safe starting point

If you need a baseline, this configuration handles most initial runs safely: r=16, α=32, a learning rate of 2e-4, target_modules="all-linear", dropout scaled to your data, and one to two epochs. Monitor the loss curves: converging training and validation loss means the configuration is sound. If validation climbs while training drops, reduce the rank or the epoch count. These few numbers decide ninety percent of the result.