Academy

A practical checklist for picking an LLM for your feature

Choosing a model is less about leaderboards than about knowing your own need precisely. Six simple steps you can follow today.

The Felesh team 4 min

Choosing a language model for a specific feature is less about leaderboards than about knowing exactly what you need. Here is a practical checklist to follow.

1. Define the job precisely

Before anything else, define exactly what the model must do and which specific capability is decisive for the task. An intent classifier does not need deep reasoning; a text summariser needs strict adherence to format. Once you identify the core capability, half the decision is made.

2. Set the bar by the cost of failure

For each capability, define your required threshold by the cost of failure, not by the highest possible benchmark score. If an error is merely annoying, a lower threshold suffices. If an error costs money or breaks trust, the requirement is critical. This prevents paying for compute you do not need.

Three features, three barsThree features with the capability that decides each one and the cost of a wrong answer. Routing a message needs the exact label, and a wrong desk costs a minute: the bar is low. Summarising a visit needs its format kept, and a wrong fee on file breaks trust: the bar is high. Drafting a reply needs the right tone and facts, and a person reads it first: the bar is medium.featurebar we setroute a messageneeds: the exact labela minute lostlowsummarise a visitneeds: the format keptwrong fee on filehighdraft a replyneeds: tone and factsa person reads itmedium
Each feature has one decisive capability, and its cost of failure sets how high the bar must be.

3. See what the surrounding system guarantees

The model does only part of the job. Before demanding perfection from the model itself, examine what the surrounding system already prepares, checks, or corrects. If the output is validated downstream, the model may not need to be flawless on its own.

A check between the model and the calendarThe model reads “move my Tue 09:30 visit to Thu” and writes the action reschedule, which the check does not allow. The check allows book, move and cancel, so it holds the output: the bad action is outlined and marked not allowed, and the request goes back to the model or a person. The model writes move, the check passes it, and the calendar shows Thu 09:30.model outputvisit: Tue 09:30action: “reschedule”action: “move”to: “Thu”checkallowed:bookmovecancelnot allowedback to the model, or a personcalendarthe visitThu 09:30
A check after the model stops a wrong action before it reaches the calendar, so the model does not have to be flawless.

4. List the practical needs

Next, outline the practical constraints: required context length (remembering that advertised long context does not always mean reliable recall), output format and whether you require structured output, latency budgets, cost ceilings, and hosting or privacy rules. Any single constraint here can rule out a candidate on its own.

5. The smallest model that does the job

Among the models that meet all of the above criteria, make the smallest one your default. A larger model is only justified when the smaller one clearly falls short on a decisive capability. Smaller usually means faster and cheaper.

Four candidates against three limitsFour models for a 40-page tender summary, which needs about 60k tokens of context, JSON output and an answer under 2 s. Mid B has a 16k context, so it is out. Large D returns free text, so it is out. Small A (128k, JSON, 0.8 s) and large C (200k, JSON, 1.9 s) clear all three; the smallest of them, small A, is the default.candidatecontextoutputlatency40-page tender60kJSONunder 2 ssmall · A128kJSON0.8 smid · B16kJSON0.9 slarge · C200kJSON1.9 slarge · D128kfree text1.1 ssmallest left: small · A, 128k, JSON, 0.8 s
A candidate that fails one limit is out; of those left, the smallest is the default.

6. Test on your own data

Before making a final decision, test the candidates on a few dozen real examples from your own workload. No public leaderboard replaces this specific test; it is the only way to know which model is genuinely right for the job.

Your own examples, read row by rowFour examples from the feature’s own messages, each with its expected answer and what two candidates answered. Move Tue 09:30 to Thu: expected move, both answer move. Why was I charged twice: expected bills, model A answers info, model C answers bills. Cancel all, I moved abroad: expected a person, model A answers cancel, model C answers a person. Is parking free after 18:00: expected info, model A answers info, model C answers bills.exampleexpectedmodel Amodel CMove Tue 09:30 to Thu.movemovemoveWhy was I charged twice?billsinfobillsCancel all, I moved abroad.personcancelpersonIs parking free after 18:00?infoinfobillsmodel A 2 wrong, model C 1 wrong
Run the candidates on your own examples: where one gives a wrong answer, the examples decide, not the leaderboard.