Academy

Stop ranking LLMs, start profiling them

A single number on a leaderboard won't tell you which model fits your job. A multi-dimensional profile of capabilities will.

The Felesh team 6 min

Every time a new language model ships, the first question is usually: “where is it on the leaderboard?” That question is natural, but misleading. A single rank compresses dozens of distinct capabilities into one number — and in that compression, exactly the thing that matters for your decision gets lost.

Models are like humans: someone who is strong at maths is not necessarily a good writer. Asking “who is the smartest?” is almost meaningless; the right question is “for this specific job, who is the better fit?” Instead of ranking, we should profile.

What ranking hides

A model might excel at mathematical reasoning yet slip when generating structured output. Another might follow precise instructions perfectly, yet forget details buried in the middle of long documents. If you only look at the overall rank, these variances remain invisible — until they surface in production as an expensive error.

A rank hides the dimension that decides the pickA leaderboard ranks model X first and model Y second. Across six dimensions, model X slips on long-context recall and on latency and cost, while model Y slips only on structured output. The rank hides which of those matter for your job.leaderboardmodel Xrank #1model Yrank #2one number,ranks X firstdimensionmodel Xmodel Yinstruction followingholdsholdsstructured outputholdsslipslong-context recallslipsholdsreasoning depthholdsholdsrefusal behaviourholdsholdslatency and costslipsholds
The rank puts model X first, but on the profile each model slips where the other holds, so your job decides.

The problem with the leaderboard is not that it is wrong. It is that it does not ask your question. Your question is always specific: how well does this model do my job?

Instead of a rank, a capability profile

Profiling means scoring each model across several independent dimensions rather than a single axis. In practice, the dimensions that matter are usually these:

  • Instruction following — when you set an explicit constraint, how precisely does it hold to it?
  • Structured-output stability — does it produce valid, repeatable JSON or the requested format?
  • Long-context recall — does it find a detail buried in the middle of a long input?
  • Reasoning depth — on multi-step problems, does it sustain the chain of logic to the end?
  • Refusal behaviour — where does it correctly say “I don’t know,” and where does it answer incorrectly with false confidence?
  • Latency and cost — does the quality it delivers justify the time and financial cost?

The key point is that none of these is general; you must measure them against your own work. Build a small evaluation set of real or near-real examples — even 20 to 30 cases are highly informative — and score each model on those exact examples, dimension by dimension.

Six dimensions, each as one scored questionSix rows, one per dimension. Instruction following asks for at most three bullets and scores one when the count holds. Structured output asks whether the reply parses as JSON. Long-context recall asks whether clause 14.2 is found in 18,000 tokens. Reasoning depth compares the final figure with the reference. Refusal behaviour asks whether the model declines a question with no answer in the text. Latency and cost asks whether the reply is inside your time and cost limit.dimensionthe probe asks, and how it scoresinstruction followingAnswer in at most three bullets.scored 1 when the count holdsstructured outputDoes the reply parse as JSON?scored 1 when json.loads succeedslong-context recallIs clause 14.2 in 18,000 tokens?scored 1 when the clause is quotedreasoning depthDoes the result match the steps?scored 1 if it equals the referencerefusal behaviourDoes it decline with no answer?scored 1 if it declines, else 0latency and costInside your time and cost limit?scored 1 only when both limits holdscore010101010101
Each dimension becomes one question your own cases answer, scored from zero to one.

A simple sketch for profiling

The idea fits in a few lines of code. Turn each dimension into a probe: a function that takes a model’s output and returns a score between zero and one. Then run those same probes across every candidate model and build a profile.

One reply, two probes, one failsOne case: the prompt asks for the delivery date from an email, as JSON. The model’s reply is cut off before its JSON closes: the delivery line holds 2026-07-03 but the closing part is missing. The structured-output probe parses it and fails it, scoring zero. The long-context recall probe finds the date in the email and scores one.one casepromptfind the deliverydate in the emailreply as JSONmodelreply”delivery”:“2026-07-03”cut off herestructured outputparses as JSON, else 00long-context recalldate found in the email1
The reply is cut off before its JSON closes, so the structured-output probe fails it while the recall probe still passes it.
# Profile, not rank: score each model across independent dimensions.
# `probes` maps a dimension name to a function that evaluates a model's
# output on one example and returns a score in [0, 1].

def profile_model(run, dataset, probes):
    scores = {trait: [] for trait in probes}
    for example in dataset:
        output = run(example["prompt"])          # one call to the model
        for trait, probe in probes.items():
            scores[trait].append(probe(example, output))
    # mean of each dimension -> that model's profile
    return {trait: sum(v) / len(v) for trait, v in scores.items()}


def valid_json(example, output):
    import json
    try:
        json.loads(output)
        return 1.0
    except ValueError:
        return 0.0


probes = {
    "structured_output": valid_json,
    "instruction_following": follows_constraints,   # your own probes
    "long_context_recall": finds_buried_fact,
    # ... whatever dimension matters for your job
}

profiles = {name: profile_model(run, dataset, probes)
            for name, run in candidate_models.items()}

The output is no longer a single number; it’s a table showing each model’s strengths and weaknesses side by side. Now the choice is an informed judgment: you pick the model whose profile matches the shape of your work — not the one that ranks higher on some arbitrary axis.

Why this makes better decisions

It has three clear benefits. First, your choice is tied to your job, not to a generic benchmark that may have nothing to do with your need. Second, when a new model ships, you do not have to rely on rumour; you run the same probes and within minutes know whether it is better for your job or not. And third, profiling surfaces weaknesses before production — where fixing them is cheap, not where it is expensive.

Where profiling loses

Profiling is not free, and it is not always the right call. Building and curating the evaluation set requires real upfront effort, and that set must be maintained as the job drifts — a profile measured against last quarter’s cases can quietly go stale. A set of 20 to 30 examples is informative but small enough to be noisy: a single-case swing moves a dimension’s score, and tuning your choice too tightly risks overfitting to the evaluation rather than the job. For a throwaway script, a one-off task, or a low-stakes pick, standing up the whole probe harness is overkill — there, a leaderboard or arena rank is a perfectly good cheap heuristic. Profiling earns its cost when the decision is durable and the failure is expensive; below that line, ranking is the honest shortcut.

One flipped case moves the score by four pointsLong-context recall measured on 24 cases, each a grid cell that passes. Case 17 flips to a fail, where the clause sits at 18,000 tokens and the model misses it, and a ring marks that cell. One case in 24 moves the dimension by about 4 points, so a small gap between two models is noise.long-context recall24 cases, each pass or failcases 1 to 24case 17 now fails at 18,000 tokensone case in 24 moves it about 4 pointsso a gap of a few points between models is noise
On a 24-case set one flipped case moves a dimension by about 4 points, so a small gap between models is noise.

The leaderboard is good for a headline. A durable engineering decision needs a profile. For those, don’t rank your models; profile them.