Snorkel raises Open Benchmarks Grants commitment to $30 million

Snorkel says it is increasing its Open Benchmarks Grants commitment tenfold, from an initial $3M to $30M, to back work on open AI evaluation. The funding is intended for researchers, domain experts and open-source teams developing benchmarks and measurement approaches. The expanded program includes support for work on safety and alignment, cybersecurity, physical AI, long-running agent tasks, open-ended outputs, dynamic settings and human uplift. Snorkel also introduced a red-team program focused on finding and addressing weaknesses in benchmarks.

The red-team effort is intended to probe issues including reward-hacking exploits, contamination, faulty verifiers and gaps in the diversity of benchmark data. A third part of the expansion is the Snorkel Research Fellowship, which offers independent evaluation researchers access to the company’s research team, domain experts, computing resources and engineering support. Snorkel says its initial grants commitment and wider research collaborations have involved teams behind benchmarks including Terminal-Bench, ARC-AGI-3 and OSWorld 2.0. The company says benchmarks funded through the program have appeared on model cards from every major frontier lab.

Snorkel argues that benchmark development needs to keep pace with advances in frontier models. Benchmarks help assess AI systems, but the company says evaluations can lose value when tests remain too simple, static or similar to one another. Models may then become overfitted to the measures, distorting the incentives those measures create for development. The company says a broader mix of benchmarks that are robust, diverse, regularly updated and independently developed can help address this problem.

The announcement also points to challenges in testing agents that work across complex settings and extended workflows. Snorkel says evaluations need to account for specialized knowledge, messy context, multimodal inputs, tools and coordination with people or other agents. It also calls for tests that assess whether agents sustain progress, recover from mistakes and adapt as goals or environments change. For AI builders, the larger commitment and red-team program mean more support for creating benchmarks and testing their weaknesses, issues that can affect how reliably evaluation scores reflect model capabilities.