Academy

Evaluate models on your own set, not the public leaderboard

Public leaderboards tell you less about your work than you think. The reliable method: build a small eval set that represents your real task, and score the candidates on that.

The Felesh team 6 min

With every new model release, our first move is usually to check its place on the public leaderboards. Yet those tables tell us far less about our real work than we think.

The problem with the public leaderboard

A single number on a leaderboard compresses dozens of distinct capabilities, which means the very capability that’s critical for your project gets hidden in the average. The more serious problem is data contamination: because public benchmarks are repeated so often, parts of them gradually find their way into models’ training data, so a high score doesn’t always mean real ability. On top of that, your task is rarely exactly what the public benchmark measures.

One score hides the capability that mattersSix capability bars from a public leaderboard: code, math, chat, summaries and translation are long, while Persian scans, the one this project needs, is short. An average line cuts across them and feeds a single overall score on the right. That one number looks strong, and the weak Persian scans row is lost inside it.what the leaderboard testswhat it reportscodemathchatsummariestranslationPersian scansaverageoverallone scoreaverage of sixyour taskstrong on average, weak on the row you need
A leaderboard averages many capabilities into one number, so the one your project depends on disappears inside it.

Build your own small eval set

The reliable method is simple: gather a few dozen real examples of the very task you intend to give the model, along with their correct answers. It doesn’t need to be large; it just needs to represent your real work and cover the hard and edge cases. Then test the candidate models on that set. It takes only a few hours, but unlike any public leaderboard it gives you a far more accurate picture of how each model performs on the real job.

What an eval set holdsFive rows from an eval set for reading Persian financial documents. Each row has a document with a note on its condition, the field to read, the correct answer a person checked, and a tag. A birth certificate, father’s name Reza Karimi, easy. A pay slip, net pay 41,250,000, easy. A bank statement with a stamp over the total, closing balance 18,430,000, awkward. A birth certificate with a handwritten date, birth date 1371/04/19, awkward. A pay slip with the scan tilted 12 degrees, net pay 9,870,000, awkward. About 27 documents in all.documentfieldanswerkindbirth certificateclean scanfather’s nameReza Karimieasypay slipclean scannet pay41,250,000easybank statementstamp over the totalclosing balance18,430,000awkwardbirth certificatehandwritten datebirth date1371/04/19awkwardpay slipscan tilted 12°net pay9,870,000awkwardabout 27 documents in all, every answer checked by a person
A few dozen real examples, each with an answer a person checked, covering the easy cases and the awkward ones.

Tie the bar to the cost of failure

Before you evaluate, decide what level you need for each capability — and set that level by the cost of failure, not by the highest score possible. If an error in a capability is only mildly annoying, the level you need is low; if its failure destroys money or the trust of your users, the level is critical. Then set a clear rule: a model passes only if it meets at least the required threshold on every necessary capability. Finally, among the models that clear that bar, choose the smallest.

A bar per capability, then the smallest that passesThree capabilities, each with the cost of failing it and the bar it needs: employer name spelling, mildly annoying, low; net pay amount, money lost, high; invented values, users’ trust lost, critical. Three candidate models are checked against every row. The small model clears the first two rows and misses the third. The mid model clears all three. The large model clears all three. The mid model is the smallest that passes, so it is the pick.capability · cost of failurebarsmallmidlargeemployer name spellingmildly annoyingnet pay amountmoney lostinvented valuesusers’ trust lostlowhighcriticala model passes only at or above the bar on every rowamong those that pass, take the smallest: mid
Each capability gets the bar its failure costs, and the pick is the smallest model that clears every one.

When the measure changes, the winner changes

This is the crux: when you measure exactly what matters for your project instead of relying on a single overall score, the ordering of models often shifts. A model that looks weaker by public evaluation may turn out to perform best on the specific capability you need; while a famous, much-discussed model may fail on your particular job. Public leaderboards only show what models do in general, whereas a dedicated eval set tells you which model fits your need. For a sound decision, you need only the second.

The ranking changes with the measureTwo ranked lists of the same three models. On the public leaderboard: flagship first, last-generation large second, mid-sized third. On your own set: mid-sized first, never inventing data; flagship second, exact on amounts; last-generation large third, less reliable overall. Lines connect each model between the lists, and the mid-sized one climbs from third to first.public leaderboardyour own set1flagship2last-gen large3mid-sized1mid-sizednever invents data2flagshipexact on amounts3last-gen largeless reliablethe order moves when the measure does
Measure what matters to your project instead of one overall score and the order of the models changes.

What this looked like in practice

We once had to choose a vision model to read Persian financial documents — birth certificates, bank statements, pay slips, and the like. So we built exactly the set described above: about two dozen real documents, each paired with a human-checked answer, covering the easy formats and the awkward ones, and scored the candidates on it.

The result rearranged our expectations. The strongest candidate was not the largest: a mid-sized model was the most consistent and never invented data, while a larger previous-generation model and a flagship several times its size each lost on overall reliability — though that flagship was the most precise on numeric amounts, exactly the kind of split that tells you to tie the bar to the capability that matters. And the decisive signal was one only our own set could surface: one model returned plausible but fabricated values — fine-looking, wrong, and invisible on any public score. For a financial task that single failure mode was disqualifying, and we saw it only because we had our own answers to check against.

A fabricated value caught by your own answersA pay slip lists gross 48,000,000, tax 4,320,000, insurance 2,430,000 and net pay 41,250,000. The model reads it and returns net pay 42,150,000: fine-looking but wrong. Beside it sits the reference answer a person checked: net pay 41,250,000. Comparing the two shows a mismatch of 900,000. No public score would have shown it.pay slipgross48,000,000tax4,320,000insurance2,430,000net pay41,250,000model readsnet pay42,150,000looks rightone field, read two ways, then comparedreferencechecked bya personnet pay41,250,000compare with our answer900,000 offplausible, wrong, and no public score shows it
A plausible but wrong value passes unnoticed until you have your own answers to compare it with.

One honest caveat, because honesty about evidence is part of the method: this was a small, curated set of about 27 documents, scored by meaning rather than exact string match, run in early 2026 on an earlier version of our pipeline that we have since replaced. It is not a production guarantee or a benchmark result — it is an illustration of what evaluating on your own data buys you.