Why AI Benchmarks Are Broken, But Rankings Survive

Overview

Artificial intelligence benchmarks are widely considered the rulers of the field, serving as both measuring sticks and dominant forces that shape research direction. However, Moritz Hardt, a computer scientist and director at the Max Planck Institute for Intelligent Systems, argues that this ecosystem is fundamentally broken.

In his 2026 book, The Emerging Science of Machine Learning Benchmarks, Hardt contends that these standardized tests are "the mistake that made machine learning." Despite this flaw, he suggests that benchmarks have driven legitimate progress largely by sheer luck—a credibility they are now losing in the era of large language models.

"The benchmarking ecosystem is broken. There’s no two ways about it." — Moritz Hardt

The History of Benchmarking Failures and Fortunes

The relationship between AI research and benchmarks has been historically volatile. The first benchmark was created in 1959, featuring digits and letters on punch cards to test character recognition scanners. It introduced the concept of splitting data into training and test sets, a practice that has since become the engine of AI progress for nearly six decades.

However, early benchmarks did not always spur growth. In 1987, the Defense Advanced Research Projects Agency (DARPA) released its first automatic speech-to-text translation benchmark. The results were disheartening; state-of-the-art AI could not outperform simple statistical approaches. This poor performance contributed significantly to an AI winter, causing research interest to wane and funding to dry up for approximately 25 years.

The pendulum swung back in 2012 with the ImageNet benchmark. An eight-layer neural network trounced other algorithms in image classification, triggering the deep learning revolution. This success cemented the benchmark as a primary driver of technological advancement, despite the underlying methodological issues that Hardt now highlights.

Why the System Is Broken

The traditional recipe for benchmarking is deceptively simple. Researchers take their data, split it into a training set and a test set, and allow model builders to do whatever they want with the training data. The test set is then used solely to compare and rank models, creating a leaderboard that declares a state-of-the-art winner.

The problem arises from the violation of the "holdout method." Machine learning textbooks explicitly warn against using the test set during development, yet the field largely ignored this advice. Researchers put these test sets on the internet, making them easily accessible for inclusion in development loops. Consequently, model builders iterate on the same data repeatedly, allowing their systems to game the leaderboard without demonstrating genuine intelligence.

This leakage has severe implications for accuracy. If a model achieves 90% accuracy on one version of a benchmark but only 73% on a slightly modified version, the metric loses its meaning. It becomes impossible to determine the true capability of the model, rendering the score misleading at best and useless at worst.

"The fascinating thing is that machine learning as a discipline just ignored the warning. It took all these test sets and put them on the internet for everyone to use." — Moritz Hardt

The Paradox of Progress

Despite these fatal flaws, benchmarks have historically driven progress. Hardt attributes this to a phenomenon of "accidental credibility." For many years, the competition for leaderboard dominance forced researchers to develop more robust and generalizable algorithms. The intense pressure to win created a friction that, by luck, aligned with genuine scientific advancement.

However, this alignment is breaking down in the age of large language models. As models become more capable of memorizing and reverse-engineering benchmark datasets, the scores they receive no longer correlate with actual reasoning abilities. Tech companies like OpenAI and Anthropic continue to advertise high benchmark percentages as proof of superior reasoning, but Hardt argues this is a misuse of data.

The disconnect is stark. A model might score 90% on a math benchmark, leading marketers to claim the AI is nearly as good as a human mathematician. Yet, when faced with a novel problem outside the trained distribution, the model may fail catastrophically. The benchmark score has become a hollow metric, reflecting the model's ability to exploit the test rather than its capacity for general intelligence.

Toward a Science of Benchmarking

Hardt’s diagnosis is not merely critical; it is a call to reform. He believes that benchmarking can be fixed if the field shifts from treating it as a dark art to viewing it as a rigorous science. This requires moving beyond simple accuracy scores and developing evaluation methods that are resistant to gaming.

Potential solutions include creating dynamic benchmarks that change over time, using human evaluation more extensively, and developing tests that measure reasoning processes rather than just final answers. The goal is to restore the link between high scores and genuine capability, ensuring that benchmarks once again serve as reliable thermometers for the state of AI.

Until such reforms are implemented, the AI community must remain skeptical of leaderboard rankings. While comparisons between models can still offer some insight, the absolute scores advertised by tech giants should be viewed with extreme caution. The era of blind faith in benchmarks is over, and the challenge ahead is to build a new standard that can withstand the pressures of modern artificial intelligence.