Academy

AI estimates — it doesn't calculate

You trusted a confident AI answer the way you'd trust a calculator. Then you discovered it was wrong. Here's why that keeps happening, and what to do about it.

The Felesh team 7 min

The calculator that isn’t

Type =2+2 into a spreadsheet. You get 4. Every time, without variation, for every person on Earth. The spreadsheet calculates — it applies a fixed rule to an input and arrives at the only correct answer.

Now ask a large language model “What is 2+2?” You will likely get 4 too — but for a completely different reason. The model has seen the question and its answer countless times in its training data, so it predicts that 4 is the most probable continuation. It isn’t doing arithmetic. It is pattern-matching.

Calculate versus predictTwo lanes for 2+2. In the spreadsheet, the cell =2+2 passes through a fixed rule that adds the numbers and gives 4, the same on every run. In the model, 2+2 = is matched against text seen countless times, and the candidates 4, 5 and the word four are weighed; 4 is by far the most probable, so it wins, with no rule behind it.spreadsheet · calculates=2+2fixed ruleadd the numbers4same on every run, for everyonemodel · predicts2+2 =seen “2+2=4”countless times45fourthe most probable continuation wins, with no rule behind it
A spreadsheet applies a fixed rule and always lands on 4; a model lands on 4 because that continuation is the most probable one.

That distinction — calculate versus estimate — is the single most useful frame for working with AI.

What “predict the next token” actually means

An LLM (a large language model, the technology behind ChatGPT and similar tools) is trained on an enormous volume of text. Its job is to answer one question over and over: given everything written so far, what word — or word-fragment — is most likely to come next?

Each output is a probability distribution over the vocabulary: this word 40%, that word 25%, another word 15%, and so on. The model samples from that distribution, appends the result, and asks again. There is no lookup table, no formula, no notion of “the correct answer.” Only: what is most probable?

One token at a timeThe text so far is: Your refund will arrive in. Asked what comes next, the model spreads probability over candidates: 3 at 40 percent, 5 at 25, a few at 15. One is sampled, 3, and appended. The question is asked again for the longer text, and the new spread is days 62 percent, hours 18, weeks 6; days is sampled and appended. It repeats until the answer ends.the text so farYour refund will arrive in3days1 · what comes after “in”?35a few40%25%15%2 · what comes after “in 3”?dayshoursweeks62%18%6%repeat until the answer ends: no lookup, no formula
Each step turns the text so far into a spread of likely next words, samples one, appends it and asks again.

This works surprisingly well across a vast range of language tasks. It fails in ways that are equally surprising — and important to understand.

Why it can be fluent and wrong at the same time

Fluency and accuracy are independent properties. A sentence can be grammatically perfect and factually false. It can sound authoritative while describing something that never happened.

An LLM doesn’t have a fact-checker sitting alongside it, cross-referencing claims against reality. It only knows that certain patterns of words tend to follow other patterns. If a confident, well-structured sentence is the most probable continuation, the model produces it — regardless of whether the underlying claim is true.

Fluent, and nothing checks itThe model’s answer, The Halden Accord was signed in Lisbon in 1887, is checked on two scales. The fluency bar fills completely, with grammar, tone and structure all ticked. The accuracy track reads not checked: no fact-checker sits alongside the model, so nothing verified the claim, and a rust break mark says so.the model’s answerThe Halden Accord was signed in Lisbon in 1887.fluentgrammartonestructureaccurate? not checkedno fact-checker alongsideit writes what is likely to appear in text, true or not
Fluency and accuracy are independent: the sentence reads perfectly, and nothing in the model checks it, so it can be wrong.

This is what the field calls hallucination: the model generates plausible-sounding content that is wrong. It isn’t lying, in the human sense. It isn’t deliberately deceiving. It simply has no mechanism to distinguish “likely to appear in text” from “true in the world.”

Research papers studying various models have documented hallucination rates ranging from a few percent to over twenty percent, depending on the task type and domain — though the numbers vary widely by model and evaluation method. The point isn’t the exact figure. It is that hallucination is an inherent property of the architecture, not a bug that will be patched away entirely.

Why the same prompt gives different answers

Because the output is sampled from a probability distribution rather than computed from a formula, adding a controlled amount of randomness to the sampling (a setting called temperature) means the same prompt can produce different outputs each time. That isn’t a defect — it is sometimes desirable for creative tasks. But it means you cannot treat an LLM like a deterministic function.

Same prompt, different answersOne prompt, When was the Halden Accord signed, is run three times in two ways. With no randomness, temperature 0, all three runs return 4 March 1887. With controlled randomness, temperature 0.8, run one returns 4 March 1887, run two returns 4 March 1878 and run three returns 12 March 1887.“When was the Halden Accord signed?”no randomness · temperature 0same answer hererun 1run 2run 34 March 18874 March 18874 March 1887randomness added · temperature 0.8different each runrun 1run 2run 34 March 18874 March 18874 March 18874 March 187812 March 1887so an AI answer is not a deterministic function of the question
With randomness added to the sampling, the same prompt can return a different date each time.

Ask it to count the words in a sentence: it might get it right, and it might not. Ask it to verify a number: it will reason around the number rather than count precisely. Ask it to recall a publication date: it may produce a plausible but incorrect one. These aren’t edge cases. They are predictable weaknesses that follow directly from the architecture.

Why it struggles with exact arithmetic

A calculator holds and manipulates exact values. An LLM manipulates tokens — text fragments that happen to look like numbers. When you ask it to multiply 847 by 293, it isn’t running multiplication. It is producing the token sequence that most resembles the output of a multiplication, based on examples it has seen. For small, common calculations, the pattern-match usually works. For larger or unusual ones, it often doesn’t.

This isn’t a solvable problem within the current architecture. It is why well-designed systems pair a language model with actual computational tools — a real calculator, a real database — when precision is needed. You don’t ask the probabilistic component to do the deterministic work.

Guess the digits, or call the calculatorThe question is 847 times 293. Alone, the model writes 248,371, which looks like a product but was never computed, and a digit is off. Paired with a calculator, the model passes 847 times 293 to it and 248,171 comes back, exact because a tool computed it.“What is 847 × 293?”model alonemodel248,371looks right, was never computedmodel + calculatormodelcalculator847 × 293248,171exact, because a tool computed it
Alone, the model writes digits that look like a product and can be wrong with nothing to catch it; given a calculator, it asks and the exact answer comes back.

The practical posture: verify, don’t trust

None of this means the technology is useless. A skilled estimator — a doctor reading symptoms, an editor reading a draft, an engineer reviewing a specification — is genuinely valuable even though their output isn’t guaranteed correct. The appropriate response to an estimate is to verify it, not to take it on faith.

The same posture applies here. An LLM can draft, summarise, suggest, and reason at speeds that were impossible a few years ago. The output is a high-quality first guess, not a signed certificate. For anything that matters — a factual claim, a calculation, a legal reference, a piece of code that will run in production — check the output against a source you trust.

What to check each output againstA table of four outputs and the check for each. A factual claim, signed in Lisbon in 1887, is checked against the archive record. A calculation, 847 times 293 equals 248,371, is checked with a calculator. A legal reference, Article 14 allows this, is checked against the text of the law itself. Code for production, retries equals 3, is checked by running the tests. The output is a first guess, not a signed certificate.the output sayscheck it againsta factual claim“signed in Lisbon in 1887”a calculation“847 × 293 = 248,371”a legal reference“Article 14 allows this”code for production“retries = 3”the archive recorda calculatorthe law’s own texta test runa first guess, not a signed certificate
Each kind of output has its own check, because the output is a first guess, not a signed certificate.

That shift in posture, from trusting to verifying, is what separates people who use AI effectively from people who get burned by it.

Where to go from here

The six limits of a language model on its own — including the knowledge boundary, the inability to act, and more — are explored in From LLMs to agents. And if the vocabulary around “AI,” “LLM,” and “ChatGPT” is still blurry, AI is not one thing builds that map first.