Academy

Common fine-tuning pitfalls and how to debug them

Most failed fine-tunes trace back to a few recurring patterns. If you know the signs, debugging becomes a simple checklist instead of guesswork.

The Felesh team 4 min

Fine-tuning can feel opaque when it fails. Fortunately, most failures trace back to a few recurring patterns, and each leaves a clear signature. Here are the most common pitfalls and how to spot them.

The loss won’t come down

If the loss stays roughly flat during training, you have three primary suspects: the learning rate is too low, LoRA is not applied to the correct layers, or the α/r ratio is too small. First, check the number of trainable parameters. If that number is unexpectedly small, your configuration is likely incorrect. If the parameter count is expected, incrementally raise the learning rate.

The parameter count decides which fix comes firstThe loss sits at 1.84 at step 500 and at 1.83 at step 1000. The first check is the trainable parameter count: 0.4M, where rank 16 on the query and value projections should give about 8M. That gap means LoRA is not on the layers it should be, so the configuration is fixed first. Only a count that matched would send you to the learning rate.flat loss, steps 500 to 1000loss 1.84 at step 500 · 1.83 at step 1000check firsttrainable params0.4Mexpected about 8MLoRA misses the layersfix target modules, then reruncount as expected:raise the learning ratecount first: the learning rate waits until the count matches
Before touching the learning rate, count the trainable parameters: 0.4M where about 8M is expected means LoRA is missing its layers.

Overfitting

If the training loss decreases while the validation loss climbs, the model is memorising the data rather than learning the underlying pattern. To fix this, reduce the number of epochs, raise the dropout rate, lower the rank, or implement early stopping. Overfitting is particularly common on small datasets.

Validation loss climbs while training loss fallsTraining loss falls from 1.90 to 0.36 over six epochs. Validation loss falls to 1.58 at epoch 3 and then climbs to 2.04 by epoch 6. From epoch 3 the two curves split apart, which is the model memorising the training data.training loss 1.90 → 0.36, still fallingvalidation loss 1.58 at epoch 3, then 2.04123456epochepoch 3: lowest validationthe gap widens: the model memorises the training set
Validation loss bottoms out at epoch 3 and climbs after it while training loss keeps falling: that split is memorisation.

Gibberish output

If the model produces garbled or meaningless output after training, the cause is usually one of the following: the learning rate is too high, α is too large, you have trained for too many epochs, or the training data is corrupted. Lower the learning rate first; this is the most common culprit for broken output.

The same prompt at two learning ratesThe prompt asks how to reset a password. At learning rate 2e-3 the answer is garbled: Go to to Setings ## ## the the. At learning rate 2e-4 the answer reads: Open Settings, then Security and choose Reset password. Same data and prompt; the rate is the first thing to lower.promptHow do I reset my password?learning rate 2e-3Go to to Setings ## ##pa-ss rr the the ## ww## Sett ings the thelearning rate 2e-4Open Settings, thenSecurity and chooseReset password.same data and prompt: lower the rate first
Same prompt, same data, two learning rates: the high one garbles the answer, and lowering it is the first fix to try.

Running out of memory

If you run out of memory during training, apply these mitigations in order: enable gradient checkpointing (this yields the largest saving), drop the batch size to one and compensate with gradient accumulation, use four-bit mode (QLoRA), and finally, shorten the sequence length. A combination of the first two is usually enough to resolve the issue.

Slow training

If training is unexpectedly slow, you have a few structural options: use Flash Attention on compatible hardware, use bf16 instead of fp16, and increase the batch size if memory allows. The bottleneck is often found in suboptimal settings rather than the hardware itself.

Data quality

Sometimes the metrics look correct, but the output remains poor. In this scenario, suspect the data. Manually review a sample of the training records. Check whether the formatting is exact and if the examples are high-quality and consistent. A small, clean dataset almost always outperforms a large, messy one.

The evaluation gap

The final pitfall is failing to observe the problem at all. Never rely solely on training loss. Always measure performance against a separate validation set, and qualitatively compare the output against the base model. Without this baseline comparison, you risk believing you have improved the model when you have actually degraded its quality.

From pitfall to checklist

These pitfalls have clear signatures, meaning debugging does not have to be guesswork. When a fine-tuning run fails, read the symptoms — flat loss, overfitting, or gibberish output — and investigate the specific suspects linked to them. Running through these checks resolves almost all early training problems.