Small-agent teams scale unevenly across benchmarks, study finds

A study of small language-model agent teams reports that adding calls can improve accuracy, but the size of the gain depends on the task and the way agents are organized. The researchers tested eight orchestration architectures with five instruction-tuned models in the 7–9B range, across five short-answer benchmarks and an executable-code benchmark, using budgets of up to 30 calls. The paper’s central finding is that call count alone does not predict a consistent improvement across tasks.

Between three and 30 calls, accuracy increased by as much as 17 points on the GSM8K and GSMHard arithmetic word-problem benchmarks. On ARC, GPQA and MMLU, the reported gains were no more than four points across the architectures. The authors say this gap is obscured by reporting a single average across tasks, which can hide how differently the teams perform by benchmark.

Proposer-Critic, an architecture in which a proposal is evaluated by a critic, captured the arithmetic gains and scaled most steeply there. At the largest call budget, it ranked ahead of the other architectures in aggregate on the tested results, with item-clustered intervals excluding zero. But it was among the weakest approaches on other tasks, and the study found no architecture that led across all benchmarks.

To explain the differences, the researchers divide a workflow into proposal coverage and a downstream transformation step. In their framework, an accuracy change separates into a coverage dividend—more candidate answers being produced—and a change in how effectively the workflow transforms those candidates into a result. Their analysis attributes arithmetic improvements to room for broader coverage that a critic-guided process can convert into correct answers. On the multiple-choice benchmarks, coverage either approaches saturation or additional candidates are not converted into better results.

For the open-ended code benchmark, the paper reports that generative recovery nearly disappears, leaving accuracy more closely tied to coverage. The authors also found that token costs differed by a factor of 2.1 at equal call budgets. For builders, the results argue against treating extra agent calls as a general-purpose accuracy lever: both the benchmark and the orchestration design affect whether additional candidate generation pays off.