VAmoS Energy benchmark tests voice agents on complex calls and background speech

A new paper introduces VAmoS Energy, a benchmark designed to test whether voice agents can manage multiple customer needs while handling background speech. It covers 100 simulated calls about utility billing and payment assistance, with each caller making two to four requests. The authors focus on conditions that voice agents may encounter in production, including several requests and customers who lose patience. The paper was submitted to arXiv on 29 Sep 2026.

The benchmark gives agents sixteen tools connected to a stateful Stripe billing twin and the Apache Fineract loan engine. Account access remains blocked until the caller has been verified, making authentication part of the tasks. Scenarios use public household electricity data and a policy based on Pennsylvania residential billing rules. An LLM-based verifier checks agent actions and spoken figures against explicit requirements. In a calibration run, it agreed with a code verifier on 99.1% of checks.

Across fourteen voice stacks, with each task repeated three times, completion rates ranged from 17.3% to 44.7%. Grok Voice led the results, while Gemini 3.8 Live and GPT-Live followed at about the same cost per call. The benchmark evaluates whether agents complete the calls under these combined constraints, rather than testing only a single spoken response.

Background television had a marked effect: pooled completion fell from 38.7% to 8.6%. The paper also describes a limitation in how simulated callers judge outcomes: they may accept an incorrect result after hearing the agent, because they cannot inspect its actions. That means a convincing spoken response does not necessarily show that the agent performed the requested operation correctly.

For AI builders, VAmoS Energy puts multiple requests, caller verification, tool use, spoken figures and competing speech into the same evaluation. The verifier’s combination of checks reflects the paper’s concern that evaluations should account for both what an agent says and what it does. The authors argue that voice-agent testing should assess the whole call, including spoken content, changes made through tools and responses to competing speech.