Reading benchmark claims critically

Benchmark numbers are easy to quote and hard to interpret. A product page says a model scores 70-something percent; a headline says it beats a rival by two points. Whether that tells you anything depends on details that rarely travel with the number. Use the questions below on any score, whether it comes from a vendor, a research group, or us.

What should I ask about any benchmark number?

Can the same model score differently on the same benchmark?

Yes, and the clearest published example is old enough to be well documented. In June 2023, the Hugging Face team behind the Open LLM Leaderboard compared three implementations of MMLU, a widely used multiple-choice test. The implementations differed in how they asked the question and how they turned the model's output into an answer. They reported very different scores for the same model, llama-65b.

The authors concluded that MMLU numbers from different implementations are not directly comparable. The lesson is not that MMLU is bad. It is that a benchmark name is not a method. Ask which harness, which prompt format and which scoring rule produced the number. Our glossary entry for evaluation harness explains the term.

Why is one headline metric not enough?

Accuracy is only one property people care about. Stanford's Holistic Evaluation of Language Models (HELM) argued for measuring several at once, and reports seven metrics: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Across 42 scenarios and 30 models, its authors report that earlier evaluations covered an average of 17.9% of the core scenarios and that HELM raised coverage to 96.0%.

Whatever you think of any particular framework, the principle holds for buying decisions. A tool that is slightly more accurate but twice as expensive, or more accurate on average but unpredictable in its failures, may be the worse choice for you. Look for cost, latency and failure behaviour next to the headline number, and see how this site separates dimensions in the methodology.

What does a benchmark's design tell you about its score?

Look at how the tasks were built. SWE-bench, a widely cited coding benchmark, is made from real GitHub issues and their matching pull requests across 12 popular Python repositories, giving 2,294 problems. A model receives a codebase and an issue description and must produce a code change. The project's repository describes evaluating the generated patches in Docker containers, and lists a 500-problem subset called SWE-bench Verified.

That design is more realistic than a puzzle with a one-line answer, which is a strength. It is also Python-only and issue-shaped, which is a limit: a high score says little about, say, a Java service or a greenfield project. The original paper's abstract reported that the best model it tested then, Claude 2, resolved 1.96% of issues, a reminder that headline figures date quickly. Read a benchmark's scope as carefully as its score.

What does a careful read of a claim look like?

Here is an illustrative example. It uses an invented product and invented numbers, and it is not a statement about any real tool.

None of those questions is hostile. Most vendors can answer them. The ones who cannot, or will not, have told you something too.

What are the warning signs of a weak benchmark claim?

• A single number with no version, date or settings.
• A win described as large without any statement of run-to-run variation.
• Different tools shown with scores from different benchmark versions or different harnesses.
• A very high score on a long-public test, with no account of how overlap with training data was checked.
• A benchmark whose tasks and scoring rules cannot be inspected.
• A comparison where the vendor's own product was tuned and the rivals were run with defaults.

What are standards bodies saying about benchmark practice?

In January 2026 the US National Institute of Standards and Technology announced NIST AI 800-2, an initial public draft titled "Practices for Automated Benchmark Evaluations of Language Models", from its Center for AI Standards and Innovation. According to NIST's release, it sets out preliminary practices in three stages: defining objectives and selecting benchmarks, implementing and running them, and analyzing and reporting results. NIST notes that such evaluations cannot meet all evaluation objectives. We read the announcement, not the full draft, and the document may have changed since; check NIST for the current status.

What will we publish with any result?

When we publish benchmark results, we will publish the task set, the conditions, the sample size, the number of runs, the version tested, the date and the limitations, and we will label who made each claim. Until then, our benchmarks page lists the protocol requirements and contains no results. For how we label claims, see vendor claims vs. verified findings; for choosing a tool yourself, see choosing a coding assistant.