Benchmark contamination explained
What it is, how it happens, how researchers have tried to measure and avoid it, and why it can make a public benchmark score misleading.
By Adam Green9 min read
Imagine a student who has seen the exam questions and their answers a week before the test. A high mark would not tell you whether they could solve new problems. Benchmark contamination is the same problem for AI systems, and it is one reason a public score can overstate what a tool will do for you.
What is benchmark contamination?
A benchmark is a set of tasks used to compare AI systems. If a model was trained on data that includes those tasks or their answers, it may do well because it has seen them, not because it can solve new problems of the same kind. A 2024 survey of the field defines contamination as a model inadvertently absorbing evaluation-benchmark information from its training data, which makes evaluation results inaccurate or unreliable.
It is a close cousin of overfitting in machine learning, and it is distinct from deliberate cheating. Most contamination is a by-product of how large models are built: from enormous collections of public text and code.
How does contamination happen?
- A test is publishedTo be useful, a benchmark's questions and often its answers are posted online so others can reproduce results.
- Copies and discussion spreadMirrors, forum threads, tutorials and solution write-ups repeat the material on other sites and in other repositories.
- Training data is collectedWeb-scale collection pulls in whatever is public. Separating test material from training material takes deliberate effort, and partial copies are hard to catch.
- The model is tested on what it sawThe score reflects some mixture of ability and recall, and the published number does not say how much of each.
Has anyone measured it? What the GSM1k study found
One of the clearest attempts is a 2024 study by Hugh Zhang and colleagues. GSM8k is a popular set of grade-school maths word problems. The authors wrote a new set, GSM1k, designed to match it in style and difficulty, measured by human solve rates and the number of solution steps, and tested leading open and closed models on both.
| Finding | What it suggests |
|---|---|
| Some models scored up to 8% lower on GSM1k than on GSM8k | Part of the older score may reflect exposure to the older test. |
| Several model families showed this pattern across almost all sizes | It is not confined to one lab or one model size. |
| A model's probability of reproducing GSM8k examples correlated with its drop (Spearman r² of 0.36) | The authors read this as partial memorization, not as the whole explanation. |
| Many models, especially frontier ones, showed minimal overfitting | Contamination is uneven, and strong models can also generalize to new problems. |
Findings reported in the GSM1k paper's abstract (arXiv 2405.00332). Third-party research; not reproduced by us.
Both halves matter. The study is evidence that contamination is real and measurable, and also evidence against the lazy conclusion that all public benchmark scores are meaningless. The practical reading is narrower: public scores deserve a discount that varies by model and test, and fresh, comparable tests are how you estimate it.
How can I spot a possibly contaminated result?
- A very high score on a long-public test with no explanation of how overlap was checked.
- Strong performance on the test but weaker performance on similar new or rewritten tasks.
- No statement of the training data cut-off relative to the test's publication date.
- A test whose answers are widely discussed online.
- A result that is much better on public items than on a private or newer set from the same source.
What do testers do about it?
| Defence | How it helps | Limit |
|---|---|---|
| Keep the test private | Material cannot leak from the open web. | Others cannot inspect the tasks; you must trust the tester's method. |
| Keep adding fresh tasks | Newer tasks postdate a model's training data. LiveCodeBench does this with contest problems. | Costly to maintain; only protects models trained before the task existed. |
| Rewrite or perturb items | Shows whether performance survives changes in wording. | Rewritten items may differ in difficulty from the originals. |
| Check overlap directly | Finds exact or near copies in available data. | Needs access to training data, which outsiders rarely have. |
| Use human-validated subsets | Removes broken or ambiguous items; SWE-bench lists a 500-problem Verified subset. | Fixes item quality, not exposure; a public subset can still be seen. |
Common defences. Editorial summary of their trade-offs.
LiveCodeBench is a useful example of the fresh-task approach. It continuously collects new problems from programming contests, so that newer problems are less likely to be in a model's training data. Its authors report 400 problems published between May 2023 and May 2024 and an evaluation of 18 base and 34 instruction-tuned models. Dating each problem lets a reader check it against a model's cut-off.
What contamination is not
- It is not proof of intent. Accidental exposure and deliberate tuning look the same in the score.
- It is not all-or-nothing. Exposure can be partial, and its effect varies by item.
- It is not the only way a score misleads. Variation between runs, evaluation settings and unlike-for-like comparisons matter too; see reading benchmark claims critically.
How will this site treat contamination in its own tests?
No benchmark results are published here yet. When they are, we plan to follow the requirements on our benchmarks page: publish the tasks and conditions, state the date of each task and the versions tested, say whether tasks were written or collected after the tested models' training cut-offs where that is known, and describe the limits. Where we cannot rule out exposure, the result will say so. See the glossary for the short definition.
Frequently asked questions
Does contamination mean a model cheated?+
Not necessarily. Training data is often collected from the public internet, so test material can end up in it without anyone intending it. The effect on the score is the same either way.
How common is contamination?+
It varies. A 2024 study that wrote a fresh grade-school maths test, GSM1k, reported accuracy drops of up to 8% against the older GSM8k for some models, while many models, especially frontier ones, showed minimal signs of overfitting. So it is real, uneven and measurable, not universal.
Can contamination be detected?+
Sometimes. Testers check for overlap between test items and training data, compare performance on original and rewritten items, or use tasks created after a model's training data was collected. None of these is conclusive on its own.
Are private benchmarks immune?+
They reduce the risk, but they cannot be inspected by others, so you must trust the tester's method. Both public and private tests have limits.
What are fresh benchmarks?+
Tests that keep adding new tasks over time so that newer items are less likely to be in a model's training data. LiveCodeBench, for example, collects new problems from programming contests and reports 400 problems published between May 2023 and May 2024.
What should I do when a vendor cites a benchmark?+
Ask when the test was published relative to the model's training cut-off, whether the vendor checked for overlap, and whether a newer or private test shows similar results.
Sources
- Benchmark Data Contamination of Large Language Models: A Survey — Xu, Guan, Greene and Kechadi, arXiv 2406.04244, June 2024 (accessed October 14, 2026). Third-party survey; abstract read.
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic — Zhang and others, arXiv 2405.00332, revised November 2024 (accessed October 14, 2026). Third-party research; abstract read, not reproduced.
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code — Jain and others, arXiv 2403.07974, revised June 2024 (accessed October 14, 2026). Third-party research; abstract read, not reproduced.
- SWE-bench repository — GitHub (accessed October 14, 2026). Lists SWE-bench Verified as a 500-problem subset.
Related reading
This piece was first published on BestSuperintelligent.com. Read the original →