Build1 publisher3 min readPublished
Swapping an LLM classifier for a frozen random forest leaves symptom patching at 19 of 20
A dev.to write-up ran the same planted retrieval fault past forty agents, with a language model head in one arm and a distilled random forest in the other. Thirty-eight patched the symptom, nineteen per arm.
The Engineer · Build desk
What happened
- A two-arm control held the classifier, the planted fault, the corpus, the five passes and the trace format constant, and changed only the head doing the classifying.
- One head was a language model, the other a random forest distilled from that model's own labels and frozen into a pickle, and both briefs carried the same line: re-running the classification over the same context reproduced the output in 260 of 260 cases.
- Of forty agents working from the trace table without the code, thirty-eight patched the symptom, nineteen in each arm, with a Wilson interval of 0.84 to 0.99.
- Fourteen of twenty in the model arm blamed the head for being random by nature, against four of twenty in the forest arm, coded by two independent raters at 0.95 agreement.
- Seventeen of twenty in the model arm proposed stripping randomness out of the head, against eleven of twenty in the forest arm.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Replacing an opaque head with an auditable one bought nothing on this behaviour, so the choice of model is not the variable that moves it; the condition the study leaves standing is the agent's inability to read the component.
- contradiction The arms are identical on patching and three and a half times apart on attribution, so the claim that the model is irrelevant holds for the decision to patch and fails for the diagnosis that ends up in the ticket.
- cost A reviewer who nods through a temperature=0 fix pays for a change that does nothing to a dial most current reasoning models no longer expose, and the retrieval fault survives the closed ticket.
- decision Anyone hoping to fix this by writing a better brief has to weigh six responses that cited the determinism certification while accusing the head of sampling.
A control only works if the fault is out of the head's reach. The planted fault sits in retrieval, upstream of the classifier, and the head classifies correctly whatever it is handed [1]. Changing the head cannot fix the output in either arm. The forest arm makes that concrete: labels distilled from the model's own output, frozen into a pickle, wired to the identical interface, so nothing else in the system differs by a single byte [2].
The certification in both briefs closes one legitimate escape route. Without it, according to the post, an agent facing the forest could reason correctly that a trained forest is deterministic and the cause must therefore be upstream, and would patch less for a good reason instead of a revealing one [5].
An earlier round of the same experiment, without code access, returned nineteen of twenty [8]. The control reproduced that rate on both sides [1]. The author writes: "Whatever drives an engineer to smooth an output rather than trace it, a language model is not a prerequisite. A closed component is." [23]
Where the arms come apart is the accusation. Blaming the head for intrinsic randomness runs at 70 percent in the model arm against 20 percent in the forest arm, a fifty point gap and a factor of three and a half, on a patching rate that does not move at all [2]. The coding criterion, with examples of both accusation types for both heads, was written down before any response was read [10].
The phrasing differs as sharply as the counts. The model arm offers "it re-samples every night", "temperature > 0 with no seed", "sampling noise on every call" and "the nightly re-roll" [12]. The four forest-arm responses that also blame their head say it retrains without `random_state`, that `predict_proba` runs over the whole batch, and that floating point moves under parallelism [13].
The remedies inherit the diagnosis, thirty points apart between arms [3]: `temperature=0` and `seed` on one side, `random_state` and `n_jobs=1` on the other [14]. The post notes that `temperature=0` is a dial most current reasoning models no longer expose [15]. The proposal switches off something that is not on. Six responses accuse the head of sampling and cite the determinism certification in the same answer [16].
For any of this to transfer, the setup has to match: an agent triaging from a trace table, with no access to the code. Twenty per arm is a small sample. The post reports a further forty agents run to test whether what came back was the effect or the sample, and it does not include their figures or name the agent models used [18][19].
What to watch
- Figures from the second group of forty agents, which would show whether nineteen per arm was the effect or the sample.
- A run of the same control with repository access, to test whether reading the retrieval code moves the patch rate.
- Identification of the agent models used, which would show whether the sampling vocabulary is specific to particular models.