AgentHorizon tests AI judges on long computer-use tasks

Researchers have introduced AgentHorizon, a benchmark designed to test automated judges on extended computer-use tasks. The paper was submitted to arXiv on 8 Oct 2026. Its focus is a gap the authors identify between agents completing complex tasks and judges reliably assessing those completions across multiple applications. Such judges are increasingly used to decide whether an agent succeeded, including in settings where people are not involved in evaluation.

AgentHorizon contains 1,373 computer-use tasks, each pairing an instruction with a trajectory, and draws on 166 hours of human-recorded activity across three operating systems. A trajectory records the sequence of screenshots and actions taken while completing a task. The authors note that a sequence can look finished while violating the instruction or causing an unwanted side effect. To test whether judges detect such problems, the benchmark uses trajectories for related instructions and swaps the instructions to create negative examples. This design asks judges to distinguish successful work from a similar-looking trajectory that fulfills an incompatible request.

The researchers describe three benchmark splits: AgentHorizon, or AH; AgentHorizon-Simple, or AH-S; and AgentHorizon-Development, or AH-D. They evaluated eleven judges in two ways: by giving them full trajectories, and by using them as coding agents through five agent harnesses. The direct trajectory input could include as many as 300 screenshots and actions. Their best-performing agentic judge, GPT-5.5, reached 80.9% balanced accuracy on the AH subset. The paper also reports that tool use helped some models but reduced performance for open-weight models.

For builders using computer-use agents in training or evaluation, the results show why judging success needs to be tested separately from task execution: performance varied sharply in accepting valid trajectories and rejecting failed ones. The authors say judges need to find and verify evidence of completion that may be hidden in long interaction histories. That makes AgentHorizon relevant to teams assessing whether an agent followed a user's instruction, rather than merely whether its recorded activity appears complete. The benchmark provides a way to evaluate that distinction across its task pairs and splits.