Study tests what pretraining and midtraining contribute to reward learning

A paper by Chiwun Yang and Xiaoyu Li, submitted to arXiv on 29 Sep 2026, studies how pretraining and midtraining affect a model’s ability to learn from rewards. The authors focus on a limitation of reward feedback: it can indicate that an answer is right without specifying the computation needed to answer new inputs. Their work asks what information and capabilities must be available before reward adaptation can work effectively.

The paper analyzes sequential state computation and contextual memory, where different strategies can produce the same results on training rewards but disagree on new cases. It argues that observations about the underlying source, independent of the task reward, can resolve this ambiguity. The authors also describe training paths in which models first learn source prediction and then adapt to rewards using the same parameters. In this account, source prediction develops execution or retrieval, while reward learning teaches how to use those capabilities for a particular task.

Experiments using pretrained Qwen2.5 checkpoints test that division of work. Across eight worlds, sequential models given correct source and first-operation supervision achieved 82.61% success, compared with 44.15% for a control using a private-random source. In a separate memory experiment, replaying memories maintained retrieval during reward adaptation. An independent confirmation across eight worlds recorded 75.32% task success, while matched training with alternative retrieval reached 49.86%. These comparisons suggest that the information supplied before or alongside reward adaptation mattered in the tested settings.

The paper also examines GSM8K and HotpotQA, separating model accuracy when reward training begins from improvement during training and final performance. The abstract does not report benchmark scores, so it does not establish a numerical result for either dataset. For AI builders, the study highlights a distinction between rewarding correct outputs and equipping a model with the information or operations needed to produce them on unfamiliar inputs. Its results concern the tasks and experiments described in the paper, rather than establishing that the same training recipe will work across other models or applications.