Watch one of these runs end to end and the shape is familiar to anyone who has managed a capable junior with a deadline. The agent takes a base model and a benchmark, settles on a fine-tuning method early, then does hours of genuine work inside that choice: preparing data, launching training, diagnosing technical problems, repairing failed runs, picking a checkpoint [3][12]. Averaged across the study, that is about 3.8 parameter-updating runs per trajectory, 5,111 spread over 1,338 [1][4], and the plan it opened with is almost always the plan it closes with [7].
Memory, skills and feedback took the HumanEval gain from 19.4 points to 40.8 points, a little more than double, since the score moved from 22.0% to 41.4% without the addition and from 22.0% to 62.8% with it [6][11][3]. Those same additions left the decision to change strategy exactly where it was [11]. Better tools make a technician faster; they do not make the technician question the work order.
That distance is the one between what gets pitched and what is being done. A model that builds its successor is doing strategy work: switching training method, adding or dropping a stage, changing the data type, redirecting the compute that is left [3]. What the numbers document is execution depth. An average benchmark score going from 10.4% to 23.0% [5] is 12.6 points of steady grinding [2], and it says nothing about whether the grinder would have taken a different route had the first one been wrong [8].
The finding I would carry into a vendor conversation is the one about taste. Claude Code mostly reached for full-parameter supervised fine-tuning and Codex CLI for parameter-efficient fine-tuning, and which method turned up depended more on the agent than on the task [9]. Run all your post-training through one harness and part of your recipe is a procurement decision nobody wrote down; running the same job through a second harness is a cheap way to see how much of the result belongs to the method rather than the model.
This is one newsletter's account of one paper, and it is precise about the negative result: three attempted fixes, none of which moved agents toward changing strategy [10]. What it does not contain is an experiment separating cannot from was not rewarded for [5]. Agents are trained and prompted to finish, and abandoning a half-built plan reads as failure right up to the point where it reads as judgment. Until someone publishes a strategy-switch rate above the roughly 2% baseline [7] under conditions that pay for switching, the defensible read is that the ceiling has not been located rather than that it is near.
One test that survives contact with a real backlog is a two-column list: sort the last three things you delegated to an agent into plan-choice and plan-execution. If all three sit in execution, the agent is a good buy, and the review still needs a line asking what a competent engineer would have switched to and whether the trace shows the agent ever weighed it. An acceptance test that reads only the final benchmark number cannot tell a good plan executed well from a mediocre plan polished hard.
