OpenProblemBench evaluates AI on 82 unresolved science problems

Researchers have introduced OpenProblemBench, a benchmark of 82 unresolved problems from mathematics and theoretical physics. The paper presents the collection as a way to assess AI progress on scientific questions beyond established knowledge. Each problem comes with research context, assumptions and prior progress to support investigation. The benchmark is aimed at foundational theoretical science, where researchers can compare AI submissions on problems that remain open.

The researchers selected questions whose proposed solutions allow comparatively clear checks of their key mathematical or computational claims. For each submission, four evaluator models independently assess correctness, completeness and the degree of progress, without using reference solutions. This means the reported results reflect model responses judged along those dimensions, rather than a benchmark that simply checks answers against supplied solutions. The benchmark therefore combines open research questions with an evaluation process designed around the claims made in submissions.

Across seven evaluated configurations, GPT-6-Astra achieved the highest mean judged solve rate, at 14.0%. The evaluated full-size open models scored 5.5-6.7%, while Flash models scored 2.4-3.7%. These rates summarize the evaluators’ judgments of submissions, including how much progress they made, as well as correctness and completeness. The abstract reports the comparison across these configurations but does not provide further breakdowns of the results in its summary.

The paper’s case comparisons link stronger outcomes to changes in problem representation, arguments that extend beyond finite evidence, and proofs of steps needed to complete a solution. For AI builders, OpenProblemBench provides a setting to investigate model capabilities and limitations using questions drawn from the research literature. Its focus on unresolved problems also offers a way to study progress beyond tasks based on established knowledge. The authors describe the benchmark as a tool for examining AI as a contributor to foundational theoretical science, rather than as a claim that these problems have been solved.