Epoch AI finds open-ended work still limits model automation

Epoch AI evaluated six models on real tasks drawn from its own work, including graphic generation and research design, to assess how close they were to automating those jobs. The launch suite comprised 11 tasks across five categories. Epoch supplied task-relevant resources and full permissions in each model’s workspace and Google account. Models used the highest available reasoning settings and attempted assignments without further intervention. Epoch manually graded the resulting work against its own employee standards.

Claude Fable 5.1 and GPT-6 Astra performed reliably on well-defined tasks, but still fell short of automating Epoch’s work. Open-weight models lagged further behind, including on defined assignments that the leading models handled reliably. The evaluation also found that models struggled to absorb Epoch’s standards despite having reference material. They missed implicit expectations such as the organization’s visual style and the topics its audience cares about, and sometimes produced denser work than Epoch typically does.

The findings point to difficulties beyond producing an initial answer. Models could identify promising research directions, but lacked the judgment to follow through successfully. When designing experiments, they struggled to measure what they claimed to test and sometimes treated flawed results as key findings. On open-ended assignments with many possible approaches, models also tended to converge on similar ideas. These results cover tasks drawn from Epoch’s work and graded against its standards.

Epoch says its evaluation is intended to expose gaps that typical benchmarks may miss, since many benchmarks focus on easily verified tasks that represent only part of real work. Some of its assignments instead require models to gather context through tools, including Figma, Google Drive and satellite imagery ordering sites. For AI builders, the findings show why strong results on easily scored tasks alone do not establish that a system can handle work involving context gathering, organizational conventions and judgment. Epoch’s method combines structured testing with assessment of complete outputs, while its findings remain specific to the tasks and standards it evaluated.