Station agents rediscovered 62.7% of criteria from three research papers

A study of open-ended scientific work reports that agents operating in Station recovered, on average, 62.7% of the criteria associated with findings from three research papers. The authors compare that result with lower averages from Codex Multiagent-v2 and AI Scientist-v2. The paper examines whether agents can pursue research when the task is not defined by a readily available metric. It measures rediscovery criterion by criterion, rather than treating reproduction of an entire paper as a single pass-or-fail outcome.

The experiments used Station, an open-world environment where multiple agents simulate a scientific ecosystem. Researchers gave agents the main question investigated in each paper, but withheld the papers’ results and disabled web access. The tasks were built from three recent oral papers presented at ICLR. This design tested whether agents could reach findings without looking up the answers and without being given the original results.

Station’s average was 62.7% of the criteria, compared with 15.4% for Codex Multiagent-v2 and a range of 14.4% to 20.6% for AI Scientist-v2. Those figures describe the share of the original findings’ individual criteria that the agents rediscovered. The study therefore reports research coverage against a defined set of findings, not a claim that agents independently reproduced every part of the underlying papers.

The researchers added a Supervisor mechanism and periodic Meta Reflection to encourage agents to keep exploring when progress could not be judged by intermediate metrics. Their ablation and behavioral analyses found that using both mechanisms improved research coverage and continuity. The team also tested Station on two open-ended tasks without oracle papers, and reports that some agent discoveries closely matched work researchers reported after the knowledge cutoff date.

For AI builders evaluating open-ended research agents, the results make environment design and mechanisms for sustained exploration relevant factors: the study reports that adding its two mechanisms together improved coverage and continuity. The authors present the findings as evidence that a suitable environment can support meaningful progress in open-ended scientific discovery. The reported comparison is based on three paper-derived tasks, while the additional tests involved two tasks without oracle papers.