Exa introduces ATLAS benchmark for search agents

Exa said on Oct 8, 2026, it had introduced ATLAS, a benchmark that evaluates search agents on 547 wide-search tasks. The company presents the release as a response to weaknesses it sees in existing search evaluations: some can be answered from model memory, some do not resemble real research, and others lack precise or repeatable scoring. Exa says today’s best agents return correct entities more than 90% of the time, but still fail to locate a third or more of the possible entities. ATLAS is intended to monitor and reduce that completeness gap, rather than focusing only on whether returned results are accurate.

Exa defines accuracy as returning entities and their attributes correctly and tying the information to a trusted source. Completeness means identifying all available entities while also making gaps in available information visible, according to the company. For builders deploying research agents, the distinction matters because Exa says individual entities can carry economic value, while missing one critical entity can pose significant risk. The post connects that concern to work such as finding qualifying companies and reviewing firms for adverse media or patent-related prior art.

Exa says a useful benchmark should test knowledge models do not already retain; it argues that tasks more than six months old quickly lose value. The post notes that frontier-model training cycles have historically taken 3-5 months, underpinning its concern about task freshness. It also says tasks should be difficult enough to distinguish competing systems and resemble genuine user research requests rather than invented exercises. A further requirement is keeping task creation and evaluation affordable, since high operating costs can limit how often benchmarks are run.

The company outlines a common evaluation format: agents receive requests to find entities matching conditions and provide selected attributes, then are ranked by accuracy, completeness, time and cost. To preserve value, tasks need regular refreshes; Exa says they can come from researchers, user surveys or real topics drawn from an agent provider. For judging answers, the post describes two broad choices: compiling an answer key or using a rubric with an LLM judge at runtime. Exa says both approaches have historically involved major compromises, and notes that answer-key benchmarks can struggle to assemble complete, accurate answers for difficult tasks, often relying on human researchers.