AI Forecast Accuracy: Experts Underestimate Benchmark Progress

Overview

Since mid-2022, the Forecasting Research Institute (FRI) has been systematically collecting predictions from domain experts and superforecasters regarding the trajectory of artificial intelligence. A new analysis released in September 2026 reveals a consistent pattern: human forecasters have dramatically underestimated AI progress on standardized benchmarks.

The data, drawn from projects including the Existential Risk Persuasion Tournament and the Longitudinal Expert AI Panel (LEAP), suggests that while experts are often right about the difficulties of achieving artificial general intelligence, they struggle to predict the rapid pace of incremental capability gains. Conversely, forecasts regarding the immediate social and economic impacts of AI remain too speculative to draw definitive conclusions about accuracy.

Benchmark Capabilities: The Underestimation Trend

Across multiple FRI studies, the most significant discrepancy between prediction and reality lies in benchmark performance. Experts and superforecasters alike have consistently projected that AI systems would take longer to achieve human-level or superhuman results on standardized tests than they actually have.

"Forecasters dramatically underestimate AI progress on benchmarks," the report notes. This trend holds true across various domains, from language model reasoning to coding proficiency.

This underestimation bias is particularly notable because it contradicts the intuition of many policymakers who fear that AI advancement is slower than public perception suggests. The rapid iteration cycles of major AI labs appear to have outpaced the mental models used by external forecasters, who may have anchored their predictions on previous periods of slower improvement.

However, the record is not uniformly negative. Forecasts regarding specific metrics, such as Chinese model performance on the Epoch Capabilities Index, have proven to be broadly accurate. Similarly, predictions on the rate of data center growth aligned closely with actual infrastructure deployment, suggesting that while capability leaps are hard to foresee, physical scaling trends are more predictable.

Adoption and Diffusion: A Mixed Track Record

When moving beyond pure capability tests to adoption and diffusion metrics, the accuracy of experts becomes more mixed. While there is a slight lean toward underestimation, there are prominent examples where forecasters significantly overestimated the speed of real-world integration.

A notable case from a mid-2025 study involved predictions about the impact of large language models on amateur capabilities in dangerous biology tasks. The median expert view suggested a high likelihood of widespread ease-of-use improvements for hazardous biological work, a trend that has not materialized to the predicted extent.

Similarly, forecasts regarding the ubiquity of autonomous vehicles appear to be major overestimations. The median expert assessment indicated it was very likely that autonomous driving would be far more widespread by the end of 2027 than current reality dictates. These discrepancies highlight a recurring challenge: experts may overestimate how quickly regulatory hurdles, public trust, and technical edge cases slow down the deployment of transformative technologies.

Methodology and Limitations

The findings presented in this analysis cover a wide array of FRI initiatives, including the AI adversarial collaboration, the LLM-biorisk study, and the AI-cyber risk pilot. By aggregating data from these diverse projects, the institute aims to provide a rigorous, evidence-based view of forecasting reliability.

However, the authors acknowledge several critical limitations that prospective readers should consider:

• Early Resolution Bias: Many questions included in the analysis resolved early because AI milestones were reached sooner than expected. Because the study cannot include data on questions that have not yet resolved, there is an inherent selection bias toward identifying areas where progress exceeded expectations.
• Speculative Projections: For some unresolved questions, the FRI has utilized LLM projections to draw tentative accuracy conclusions. These projections are inherently more speculative and should be interpreted with caution.
• Insufficient Data on Macro-Impacts: There is currently not enough resolved data to accurately assess expert track records on macro-scale economic and societal impacts. Predictions regarding long-term GDP effects or broad geopolitical shifts remain too nascent to evaluate.

Looking Ahead

The disagreement within the expert community regarding AI’s trajectory remains stark. Estimates for when self-sufficient AI systems might exist vary by decades, with researchers like Ajeya Cotra and Timothy B. Lee disagreeing by approximately 40 years. Furthermore, economic projections range from Andrej Karpathy’s argument that AGI will have negligible GDP impact to Epoch’s models suggesting growth exceeding 30% annually.

In response to these wide variations, the FRI plans to continue publishing accuracy updates, particularly focusing on its Longitudinal Expert AI Panel (LEAP). The next installment in this series will explore methodologies for improving the relevance and precision of future forecasts, aiming to help policymakers and researchers navigate an uncertainty landscape that is evolving faster than predictive models can capture.

For now, the primary takeaway is clear: while experts are reasonably good at forecasting the physical and logistical aspects of AI expansion, they have repeatedly underestimated the rate at which software capabilities can improve. As the field moves forward, refining these predictive models will be essential for managing the risks and opportunities of rapidly advancing technology.