Build1 publisher2 min readPublished
Post-training agents pick their strategy before they run a single experiment
An empirical study of publicly released PostTrainBench trajectories finds the training strategy is fixed at step one and the whole remaining budget goes on local tweaks, with scaffolding and human hints improving execution and leaving that pattern intact.
The Engineer · Build desk
What happened
- The study analyses publicly released agent trajectories on PostTrainBench, spanning seven benchmarks, four base models and 20 distinct agent configurations.
- Across those trajectories the training strategy is fixed before the agent writes any code or runs any experiment, and the whole remaining budget goes on local adjustments inside that strategy.
- An experience-driven scaffold improved execution across the board, with reported gains on GSM8K and HumanEval, while the chosen strategy stayed exactly where it was.
- Human guidance did redirect the initial strategy, but the agent fell back into local adjustment loops as soon as training started.
- Extra inference compute paid off on the easier tasks and delivered almost no gain on the hardest one.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Execution scores can climb while the approach never gets reconsidered, so a rising eval curve inside a run is not evidence that the strategy was ever tested against the accumulating results.
- decision Thinking-token budget is the wrong dial on the hard problems; getting a different strategy means paying for another run rather than a longer one.
- capability The division of labour these results support is a human choosing the approach and the agent grinding it out, which is a usable arrangement as long as nobody budgets for the agent to change its mind.
- precedent Recursive self-improvement pitches now have a specific bar to clear, namely demonstrated mid-run strategy revision, not end-to-end completion of a post-training pipeline.
Lock-in before the first experiment narrows down where the strategy can be coming from. Nothing has been run, so nothing has been measured, and the only available inputs are the prompt and the model's prior. The sharpest evidence for that reading is the split between harnesses: Claude Code trajectories converge on full-parameter supervised fine-tuning, Codex CLI trajectories converge on parameter-efficient fine-tuning [6]. Two agents on the same task diverge systematically, which the authors take as the prior talking rather than the task [7].
This is an analysis of someone else's runs, so it is worth naming what has to hold for the pattern to appear in yours. The trajectories come from PostTrainBench, which formalised end-to-end post-training as an agent task and reported that frontier agents can complete it and lift downstream performance [13]. Transfer needs three conditions: your agent has comparable latitude over strategy, the budget is long enough that revision was actually available, and the labelling scheme would have registered a revision if one had happened. The released text describes the pattern and not the labelling procedure. It also carries the scaffold's GSM8K and HumanEval gains as bare point deltas with the numeric values absent [9], so the size of that improvement cannot be checked against what has been published.
The escalation is the useful part of the design. Three interventions were tried, and the strategy dimension moved exactly once, at the beginning, when a human moved it [10]. Interventions that produced a revision mid-run: zero [15]. The authors' conclusion is that the gap is a missing mechanism for spontaneously reevaluating strategy during execution, not a shortage of experience, guidance or reasoning compute [12].
If revision does not happen inside a run, then the run is the unit of exploration, and strategy search becomes a scheduling problem the operator owns. Anchoring on a default before looking at the data is a habit humans share; we usually call it taste.
The cheap diagnostic is a diff. Pull the strategy line out of the agent's first plan file and its last one, across tasks that differ in dataset size and label quality. If that line never changes, the opening choice was a default and the rest of the budget was tuning [4].
What to watch
- A published labelling procedure for how strategy changes were detected in the trajectories, which would let the lock-in rate be independently checked.
- Numeric deltas for the scaffold's GSM8K and HumanEval gains appearing in a revised version of the paper.
- Any harness that logs a mid-run strategy switch on PostTrainBench tasks, since that is the behaviour none of the three interventions produced.