Nine models rewrote already-optimal code in all 45 trials of an EffiBench study

The cause the Columbia paper names is the Evaluation Trap ^[2]. Every benchmark a model has been trained or scored against rewards producing an edit, and none of them rewards saying "this is already optimal" ^[5]. The paper frames that as a binary classification problem with a missing negative class ^[6]. A model that scores well on those benchmarks has learned that an optimisation request takes an optimisation as its answer, and whether the code needed one never enters the decision ^[5].

The tested fix is one sentence appended to the prompt: "Only suggest an edit if you are >90% confident it improves execution speed; otherwise, output ALREADY_OPTIMAL" ^[7]. It works because of an asymmetry in what can be proved without running anything. On slow code a model can reach 90% confidence from complexity analysis alone, and swapping an O(n squared) list.count() loop for a two-pointer sweep is an argument on the page ^[10]. On code already at its ceiling there is no equivalent static proof, so a model even slightly honest about its own confidence falls below the threshold ^[10].

Nine models on five problems is 45 trials ^[21]. At 44.4% correct abstention, the instruction turns about 20 of those trials into refusals and leaves roughly 25 edits standing on code that cannot be improved ^[22]. Prompt-only control stops there, and the write-up attributes the residual to models reporting 90% confidence they do not have ^[12]. On the degraded arm the instruction cost nothing: the edit rate stayed at 100%, with no false abstentions across all nine models ^[9].

The authors' prescription is execution. Run the before and the after, compare the clock, and let the model's opinion of its own work count for nothing ^[11]. I would add the threshold line, because it is one sentence and it does not suppress real fixes ^[9]. I would still block the merge on the benchmark, because the 55.6% of over-edits that survive the threshold arrive with a confident rationale attached ^[8].

Gemini 3 Flash Preview abstained correctly on 60% of the optimal problems, against 20% for Gemini 3.1 Pro Preview ^[14].

Before carrying 44.4% into your own review policy, look at the population it came from: five EffiBench problems, top-percentile human solutions as the optimal arm, deliberately degraded copies as the slow arm ^[3]. The write-up calls the setup small ^[24]. The abstention lever depends on a complexity argument being visible in the snippet, so for a 300-line function whose cost is dominated by I/O this design does not predict what the models do. All the figures here come from one dev.to account of the preprint.

The hand check is the part a reader can rerun. The function was a two-pointer sweep over a sorted list, O(n), each element touched once, already benchmarked ^[16]. Claude, GPT and Gemini each returned a rewrite to the prompt "optimize this for execution speed" ^[17]. A comprehension replaced the loop and measured slower ^[18]. Another rewrite added an early return that could never trigger ^[19]. The third swapped in a dict and a second pass, still O(n) but doing twice the work, under the note "this reduces redundant comparisons" ^[20].