ByteDance's self-evolved agent harnesses gain 3.11 held-out points inside a 4.75-point noise band

Held-out improvement across five evolution trajectories averaged 3.11 points ^[14]. The same write-up puts evaluation fluctuation at roughly plus or minus 4.75 points ^[16]. So the measured gain is about two thirds of the noise band ^[25]. At that resolution, a version that scores better on the visible feedback set is not evidence of a better harness.

The line counts do not reconcile either. Eighteen code harnesses added 17,111 lines, an average of 950.6 lines each ^[4][23]. Gemini is reported as the smallest edit at 1,006 lines, 55 lines above that mean ^[5][24]. Two readings survive: the 17,111 covers construction while the 1,006 counts changes made during evolution, or one of the figures is wrong. The dev.to write-up, published September 10, 2026, does not separate the phases ^[21].

Construction is the part that reliably worked. All 18 harnesses implemented an execution loop, and Tools, Lifecycle and Verification had high completion rates ^[6]. What the model starts from is a class with a while loop and three empty methods, plan_action, call_tool and is_complete ^[19]. Everything above that is generated across six capability areas: execution, tools, context, state, lifecycle and verification ^[20].

Generated code is not exercised code. Of 108 component instances in the code harnesses, 72 were observed running in real tasks ^[7], leaving 36 that were not, one instance in three ^[22]. Eighteen components never appeared in execution at all, and every one of them came from State and Memory ^[8]. Some harnesses implemented checkpoint logic, and across 26,679 recorded task trajectories there were zero checkpoint events ^[9]. Twenty-six thousand runs is a large sample in which to never checkpoint once.

In the Data tasks, 441 of 2,325 executions produced degraded commits the harness did not detect, about 19 percent ^[11][28]. The reported response was a Completion Check added to the harness ^[10].

The executor substitution costs 36.3 points, more than half the self-evaluated score ^[12][26]. Direction is not predictable: the Qwen harness went the other way on BrowseComp under a Gemini executor ^[13]. For the 68.8 on Terminal-Bench 2.1 to say anything about your stack, you would need the executor that grew the harness, its tool surface, and tasks from the same distribution ^[5].

Token numbers differ by far more than scores do. The GPT-5.5 harness used 29.3M tokens for a medal rate of 19.1, and the DeepSeek V4 harness used 208.4M tokens for 19.6 ^[17]. That is 7.1 times the tokens, and 179.1M extra tokens for half a point ^[27]. The post calls those the same performance while labelling one a medal rate and the other a score, so the comparison holds only if both are the same metric ^[17]. Across the MLE-bench experiment, token overhead varied by 19x between harnesses ^[18].