Researchers propose a framework to measure how AI agents learn from experience

A new paper proposes a way to evaluate whether AI agents improve through experience, rather than assessing only what they can do at a single point in time. Its authors argue that agents capable of diagnosing failures and learning from them need evaluations that capture changes in performance. They frame the assessment around three questions: whether later performance improves and generalizes, how efficiently capabilities are acquired, and where the improvement process breaks down. The paper calls the efficiency of converting experience into future gains on held-out tasks “agent plasticity”.

The researchers study this process in a controlled setting where agents turn prior experience into reusable artifacts that future instances can inherit. They measure agent performance at checkpoints on both training interactions and held-out interactions, while also accounting for the cost of learning. This design is intended to distinguish gains on experiences used for learning from performance on interactions that were held back. The proposed measure focuses on future held-out performance, not simply on whether an agent accumulates artifacts or improves on familiar tasks.

Across multiple environments, the paper reports markedly different improvement patterns among frontier models, even when they had comparable opportunities to learn. Some models made substantial, lasting gains, while others stayed close to or below their starting performance. Improvements on training interactions transferred only partly to out-of-distribution conditions, according to the paper. The model with the highest eventual performance was not necessarily the one that learned most efficiently, separating endpoint capability from improvement efficiency.

The authors also examine where the improvement loop fails: agents with low plasticity often do not reuse relevant artifacts, while agents that reuse them can still fail. They identify artifact quality, generalization, and application as possible limits for those agents. For AI builders, the framework suggests that evaluations of self-improving systems should track learning costs and held-out results over time, rather than relying on a single capability score. The paper presents this as a way to assess not just what agents can do, but how effectively experience makes them better.