Thinkingbox benchmark tests agents on stateful business workflows

A new paper introduces Thinkingbox, a sandbox for evaluating agents as they work with users and tools in business workflows that change persistent system state. The authors say many agent tests focus on executable environments, but consequential work also requires getting missing information, following policies and coordinating dependent tools. Their benchmark therefore checks whether an agent produces the intended state change without unwanted effects, rather than relying only on a plausible answer or valid tool call. The paper was submitted on 20 Aug 2026 and last revised on 1 Oct 2026.

Thinkingbox provides isolated sessions compatible with MCP, records execution traces and evaluates outcomes against the final backend state. Its companion benchmark, Thinkingbox-bench, contains 507 policy-conditioned workflows. The scenarios cover retail, hospitality, auto insurance, neobank internal IT, and consulting IT and HR support. Task-specific executable checks are designed to accept valid action sequences while flagging incorrect, missing or extra effects. Some tasks also assess whether the final response has required properties.

The authors report that performance declined when they measured repeated-attempt reliability, including for the strongest proprietary and open-weight models tested. Claude Opus 5 scored 66.50% on pass@1 and 47.53% on pass^20. Kimi-K3 scored 57.37% on pass@1 and 17.60% on pass^20. The paper says many unsuccessful attempts ended cleanly after valid state-changing actions, meaning orderly-looking tool activity did not guarantee task completion. The authors conclude that response-level and tool-call-level signals are poor proxies for completing the full workflow.

For AI builders, the results highlight a distinction between finding a successful sequence once and completing stateful tasks reliably across attempts. They also show why evaluations of business agents can check the resulting system state, not just the apparent quality of an answer or the validity of individual tool calls. The authors say they are releasing both Thinkingbox and Thinkingbox-bench.