Snorkel Pledges $30M to Open AI Benchmarks

Overview

Snorkel AI has announced a significant escalation in its support for the open-source evaluation community, pledging a $30 million commitment to Open Benchmarks Grants. This tenfold increase from their initial $3 million investment underscores the company’s belief that robust, open measurement tools must keep pace with the accelerating development of frontier artificial intelligence models.

The expanded program aims to fund researchers, domain experts, and independent open-source teams who are pushing the boundaries of how AI robustness is measured. By targeting areas such as safety, alignment, cybersecurity, and physical AI, Snorkel seeks to create a more resilient ecosystem for evaluating next-generation systems.

"Benchmarks have always been guideposts for – and drivers of – AI development... But when the pace of benchmark development falls behind that of frontier model development, their value... does as well."

This strategic move addresses a growing concern in the tech industry: the widening gap between the speed of AI model iteration and the maturity of the tools used to evaluate them. As models become more capable and complex, the need for dynamic, diverse, and independently developed benchmarks has never been more critical.

The Problem with Current Evaluation Methods

The rapid advancement of AI technology has exposed fundamental flaws in how these systems are currently assessed. Vincent Sunn Chen, writing on behalf of Snorkel, highlighted the phenomenon of "benchmaxxing," where models become overly optimized for static, simple, and correlated benchmarks. This overfitting is comparable to a student studying exclusively for a predictable test, leading to inflated scores that do not reflect true capability or safety.

When benchmark sets become too easy to game, they distort the incentives for model developers. Instead of building genuinely robust and aligned systems, researchers may focus on exploiting specific evaluation loopholes. This compromises the integrity of progress metrics and can obscure real-world risks associated with deploying advanced AI.

To mitigate these issues, Snorkel argues for a shift toward benchmarks that are more diverse, continuously updated, and independently verified. The goal is to establish a robust, open layer for evaluation that can withstand the pressure of intense optimization.

Key Components of the Expanded Program

The $30 million pledge is not merely a cash infusion but a comprehensive restructuring of how open benchmarking is supported. The initiative is built around three primary pillars designed to foster innovation and maintain rigorous standards.

Funding the Next Wave of Benchmarks

The expanded grants will prioritize support for new benchmarks and measurement methodologies in critical, under-explored areas. These include:

Safety and Alignment: Ensuring models adhere to human intentions and avoid harmful outputs.
Cybersecurity: Testing resilience against adversarial attacks and malicious prompts.
Physical AI: Evaluating robots and agents that interact with the real world.
Long-running Agent Work: Assessing sustained performance over extended workflows.
Open-ended Outputs: Moving beyond multiple-choice questions to nuanced quality judgments.
Human Uplift: Measuring how AI assists rather than replaces human decision-making.

Open Benchmarks Red Team

Recognizing that benchmarks themselves can have vulnerabilities, Snorkel is introducing a dedicated Red Team program. This initiative will help builders identify and address weaknesses in their evaluation tools. Common issues targeted include:

Reward-hacking exploits where models find shortcuts to high scores without genuine competence.
Data contamination risks that compromise the validity of training and testing splits.
Broken verifiers that fail to accurately grade complex outputs.
Distributional and diversity gaps that limit the generalizability of results.

Snorkel Research Fellowship

To support independent inquiry, the fellowship program will provide researchers with access to Snorkel’s internal team, domain experts, compute resources, and engineering support. This collaboration is designed to lower the barrier to entry for high-quality evaluation research, allowing academics and independent teams to operate with the resources previously reserved for large labs.

Historical Context and Impact

Snorkel’s journey into open benchmarking began with a $3 million commitment that yielded substantial results. The initial grants supported teams behind notable projects such as Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, and Senior SWE-Bench. These partnerships have been instrumental in designing evaluation methodologies and scaling quality control processes.

Since its launch, benchmarks funded by the Open Benchmarks Grants program have appeared on the latest model cards from every major frontier laboratory. They have helped index progress, measure alignment, and guide the trajectory of AI development. This track record provided the confidence for Snorkel to scale its investment significantly.

Bridging the Capability Gap

Closing the evaluation gap requires addressing three core axes of AI capability advancement: environment complexity, autonomy horizon, and output complexity.

Environment and Input Complexity

Real-world operating environments are far messier than typical benchmark settings. Future benchmarks must capture specialized knowledge, multimodal inputs, complex toolsets, and the need for coordination with humans and other agents. Static, clean datasets fail to represent the chaotic nature of actual deployment scenarios.

Autonomy Horizon

A critical metric for agent reliability is how long they can operate before performance degrades. Snorkel emphasizes the need for benchmarks that test sustained progress across extended workflows. These evaluations must determine whether agents can maintain context, recover from mistakes, and adapt as goals and environments shift dynamically.

Output Complexity

As AI systems produce more intricate deliverables, evaluation methods must evolve accordingly. Open-ended outputs require nuanced judgments of quality, usefulness, and risk handling. Simple binary verifiers are insufficient for assessing the utility of complex code, legal documents, or creative assets.

Advancing the Empirical Science of AI

Snorkel posits that as AI capabilities become more emergent than engineered, the field must adopt a more empirical scientific approach. Benchmarks serve as controlled experiments in this framework, allowing researchers to study what models can do and whether they operate within intended constraints.

Recent developments, such as OpenAI’s efforts involving thousands of agents in mathematics and documented covert coordination in Hugging Face incidents, highlight the necessity for new evaluation methodologies. Progress depends on granting researchers the freedom to develop novel questions and measurement methods.

By investing heavily in open benchmarks, Snorkel aims to ensure that the field can inspect methods, reproduce findings, and strengthen tasks continuously. This collaborative, adversarial maintenance is essential for keeping benchmarks trustworthy as the frontier moves rapidly forward.