Rewriting scaffolded agent runs before fine-tuning lifts a 27B model to 74.2% on Terminal-Bench 2

Raw harness traces made Qwen-3.8-27B worse on Terminal-Bench 2. Trained directly on its own successful runs, the model finished 3.6 points below the untuned version's 57.0% pass@3 ^[9][20]. The paper is arXiv:2610.02826, from IntelligenceLab and the University of Maryland ^[2]. A dev.to post summarising it says the model learned the harnesses' surface artifacts instead of the terminal mechanics underneath ^[8]. Some traces leaned on StateM's explicit transition assertions. Others leaned on self-reflection prompts or leaked evaluation scripts into the bash history ^[6]. A run that solves a Linux task by reading the internal verifier script is a fine record of how to read verifier scripts. Train on it, the post argues, and the model expects that leak in production, then falls apart in a bare bash shell ^[24].

The heavy harnesses still found the solutions ^[4]. StateM handled structured multi-phase setup tasks that needed state kept across interruptions. The reflection harness handled trial-and-error debugging ^[5]. Recursive Self-Rewrite splits the job in two: discovery under whatever harness works, then execution under a vanilla one ^[19]. The same base model plays all three roles ^[13]. Each successful trace goes through the same loop:

1. The planner reads the trace and writes a runbook of milestones, state checks and recovery paths. Its instructions say to describe validation procedures without copying output deliverables or hardcoded solutions ^[10]. 2. The critic audits the runbook against the public task description. It rejects verifier leakage, direct answers and harness-specific syntax, and a rejected runbook goes back for another rewrite ^[11]. 3. The executor runs the task from scratch in a fresh, isolated sandbox under vanilla Terminus 2, with the runbook as private guidance ^[12]. 4. The runbook is stripped from the saved trajectory ^[12].

Step 4 is the part I would copy. The runbook shapes generation but never reaches the training record, so the student sees only bash actions that worked with no harness around them ^[12]. Terminus 2, the execution harness, was also one of the three discovery harnesses ^[3].

Against raw-trace training, the rewritten data gained 20.8 points on Terminal-Bench 2 ^[21]. The post says it beat both the base model and direct SFT on every benchmark tested ^[1]. Software Terminal-Bench 100 doubled from 3.0% to 6.0% pass@3 ^[16]. Process reward on Long-Horizon Terminal-Bench rose from 0.21 to 0.29 ^[17]. At 6.0%, the model still misses 94% of that suite's tasks across three attempts ^[22].

These are the authors' numbers on the authors' tasks. They describe your agent only if production looks like a general terminal loop and the work looks like multi-step engineering tasks, because every reported score comes from a Terminal-Bench variant ^{[9][15][16][17]}. The post does not say whether the roughly 3,000 training tasks overlap the evaluation suites ^[3]. Both training sets came from that pool ^[3][14]. Any overlap would affect both, so I would trust the gap between the two methods more than either absolute score.