What is an AI benchmark, and should you trust the leaderboards?

What a benchmark is

A benchmark is a standardised test for AI models. Give every model the same set of tasks, score how many it gets right, and line the scores up. The point is comparison: a single number that lets you say model A beats model B at a particular skill. The 2026 names you will run into sort into a few families.

Coding and agentic work

• SWE-bench Verified hands the model real software bugs from real projects and asks it to fix them end to end. A human-checked subset, so the tasks are known solvable. When a vendor claims a coding score, this is usually it.
• Terminal-Bench tests “agentic” work: whether a model can complete multi-step jobs in a command line, taking a sequence of actions rather than just answering.

Knowledge and reasoning

• MMLU is a broad general-knowledge exam across many subjects, long the default yardstick. GPQA Diamond is graduate-level science questions.
• Humanity’s Last Exam is deliberately brutal: extremely hard questions across every discipline. The best models score around 35 per cent where human experts average around 90. It exists because models had started topping the easier tests.
• ARC-AGI tests fluid reasoning on novel puzzles the model cannot have memorised - closer to raw problem-solving than recall.

Computer use

• Online-Mind2Web tests whether an agent can actually complete tasks across live websites - shopping, forms, travel bookings - rather than on a frozen snapshot. It was built to catch agents that look good in the lab and fall over on the real web.

Why an owner should be sceptical

Here is the uncomfortable part. A leaderboard position is a starting shortlist, not an answer. Four reasons to treat the headline numbers with care.

• They are usually self-reported. The flashy figures come from the vendor’s own launch materials, run on its own setup, with the flattering numbers up front and the weak ones quietly omitted. One 2026 model launched with a wall of impressive scores; independent testing a day later found a hallucination rate north of 80 per cent on a benchmark the press release skipped.
• They run on different harnesses. The scaffolding around a model - how it is prompted, how many attempts it gets, what tools it is given - varies between labs, so two “SWE-bench” scores are often not strictly like-for-like.
• They can be contaminated. Benchmarks are public, so their questions circulate online, and a model trained on a big slice of the web may have absorbed them. When that happens the model is partly remembering answers rather than working them out, and the score overstates real ability.
• They rarely predict your results. This is the big one. A benchmark measures performance on its tasks, not yours. A model that tops a coding chart may be mediocre at drafting your customer emails or reading your invoices.

The research has a name for this. The 2025 paper behind Online-Mind2Web called it an “illusion of progress” - agents that looked strong on static tests failed once the websites were live. The lesson generalises: a benchmark score is evidence, not proof.

What to do instead

Do not throw the leaderboards out. Use them for what they are good at, and add the step that matters.

• Use leaderboards to build a shortlist. They are a fine first filter for which two or three models are worth your time.
• Then run your own small eval on your real tasks. Take ten to thirty actual jobs from your business - the emails, summaries, classifications or document reads you genuinely need. Write down what a good answer looks like. Run your shortlisted models against them.
• Pick the model that wins on your work, not the one that wins on a public chart. If the second-place model is better at your specific job, use the second-place model.

This does not need a data-science team. A spreadsheet, a couple of dozen real examples and an afternoon will tell you more about which model to buy than any leaderboard, because it tests the only thing you care about: your work.

Honest limits

• Benchmarks are not useless. They genuinely separate strong models from weak ones at a broad level, and a model that is bad across the board on public tests is unlikely to surprise you on your tasks.
• Your own eval is not perfect either. A small test set has blind spots, so treat it as a decision aid, not gospel. And the picture moves - new models and benchmarks arrive constantly, so today’s leader is not a permanent fact.

The one-line version

An AI benchmark is a standardised test that scores a model on a task - SWE-bench Verified for coding, Terminal-Bench for agentic work, MMLU, GPQA and Humanity’s Last Exam for knowledge, ARC-AGI for reasoning, Online-Mind2Web for computer use. Be sceptical: scores are self-reported, run on different harnesses, can be contaminated, and rarely predict your results. Use leaderboards to build a shortlist, then run your own small eval on your actual tasks and pick the model that wins on your work.