Build1 distinct publisher2 min readPublished
An empirical study of publicly released PostTrainBench trajectories finds the training strategy is fixed at step one and the whole remaining budget goes on local tweaks, with scaffolding and human hints improving execution and leaving that pattern intact.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Lock-in before the first experiment narrows down where the strategy can be coming from. Nothing has been run, so nothing has been measured, and the only available inputs are the prompt and the model's prior. The sharpest evidence for that reading is the split between harnesses: Claude Code trajectories converge on full-parameter supervised fine-tuning, Codex CLI trajectories converge on parameter-efficient fine-tuning [6]. Two agents on the same task diverge systematically, which the authors take as the prior talking rather than the task [7].
This is an analysis of someone else's runs, so it is worth naming what has to hold for the pattern to appear in yours. The trajectories come from PostTrainBench, which formalised end-to-end post-training as an agent task and reported that frontier agents can complete it and lift downstream performance [13]. Transfer needs three conditions: your agent has comparable latitude over strategy, the budget is long enough that revision was actually available, and the labelling scheme would have registered a revision if one had happened. The released text describes the pattern and not the labelling procedure. It also carries the scaffold's GSM8K and HumanEval gains as bare point deltas with the numeric values absent [9], so the size of that improvement cannot be checked against what has been published.
The escalation is the useful part of the design. Three interventions were tried, and the strategy dimension moved exactly once, at the beginning, when a human moved it [10]. Interventions that produced a revision mid-run: zero [15]. The authors' conclusion is that the gap is a missing mechanism for spontaneously reevaluating strategy during execution, not a shortage of experience, guidance or reasoning compute [12].
If revision does not happen inside a run, then the run is the unit of exploration, and strategy search becomes a scheduling problem the operator owns. Anchoring on a default before looking at the data is a habit humans share; we usually call it taste.
The cheap diagnostic is a diff. Pull the strategy line out of the agent's first plan file and its last one, across tasks that differ in dataset size and label quality. If that line never changes, the opening choice was a default and the rest of the budget was tuning [4].
Ranked by verification strength, evidence, and original report placement.
An arxiv paper titled 'What is Missing from AI Post-Training AI: An Empirical Analysis' argues that current discussion of AI-for-AI conflates two distinct capabilities, and that the missing strategy-level capability is the actual bottleneck for automated AI R&D.
The paper defines execution-level capability as iterating within a selected strategy (fixing bugs, tuning hyperparameters, reformatting data) and strategy-level capability as revising the high-level judgment about what to try next as experimental evidence accumulates.
The analysis covers a large corpus of publicly released agent trajectories on PostTrainBench, spanning seven benchmarks, four base models and 20 distinct agent configurations, attributed to Rank et al., 2026.
The trajectory-level analysis finds the training strategy is locked in at the very beginning, before the agent writes any code or runs any experiments, and the entire remaining budget is spent on local adjustments within the selected strategy.
The lock-in does not respond to task differences: the same agent converges to highly similar strategies across different tasks, while different agents anchor on different defaults.
Claude Code trajectories converge on full-parameter supervised fine-tuning and Codex CLI trajectories converge on parameter-efficient fine-tuning; the proportions stated in the paper's text are not present in the released HTML supplied.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · September 5, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Superpowers makes spec-driven work a precondition, then ships it to twelve harnesses1 distinct publisher
build
2,513 tool calls, zero refactorings: what agents actually do when you ask them to refactor1 distinct publisher
product
ChatGPT Work's real ask is your Slack, and somebody has to say yes on everyone's behalf1 distinct publisher
build
Claude Code walks the whole process table to inherit one shell's environment1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Specific but single-origin
The preprint is unusually precise about its own scope -- seven benchmarks, four base models, 20 configurations, all from trajectories PostTrainBench had already published -- and that public corpus is what makes the lock-in claim auditable at all. The weak spot is arithmetic. In the HTML we read, the sentence about how many Claude Code runs choose full-parameter fine-tuning has no number in it, and the GSM8K and HumanEval gains appear as bare units. A reader can see the pattern clearly, but the missing numbers keep its size unclear.
Confined to the benchmark
The scale claim here applies to a benchmark, not a deployed practice. Two shipped coding agents are driven through a post-training pipeline inside a research harness, across 20 configurations, and that is the full extent of the usage on record. Nothing in this reporting shows an agent-designed fine-tune in production, and the paper makes no such claim.
Slightly ahead of its corpus
The argument pushes against the AI-for-AI story rather than riding it, which is the rarer direction and keeps the gap small. Where it stretches is in generality: the conclusion is stated about agents as such, while the measurement is coding agents on post-training tasks in one benchmark, with the percentages that would anchor the strongest sentence missing from the published page.
Academic, with one self-interested figure
No vendor benefits from this framing. The authors did build the journal-plus-skill-library-plus-evaluator scaffold whose execution gains they report, so the result that flatters their own contribution is also the one whose numbers the released HTML drops. Against that, the conclusion cuts at the tools of two labs whose agents supplied the trajectories, and it argues down the recursive self-improvement expectation the field has been leaning on.
Moderate, pending anyone else's reading
One preprint, no replication, and no answer from either lab whose agent was analysed. The trajectory corpus is public, so the lock-in finding could be checked by any group willing to download it; until that happens the claim rests entirely on how these authors read it, and two of their numbers cannot be read at all.