Loading today’s stories
Build1 distinct publisher3 min readPublished
A permutation gate rejected an edit that fixed four tasks and broke one, then rejected weaker edits after the corpus grew to 40, because detection depends on how many tasks an edit moves rather than how many you own.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Start with what the gate actually computes. The permutation test takes the observed mean delta, shuffles the task labels 1,000 times, and counts how often chance alone produces a delta at least as large. For the v0.1.0 edit, 23 percent of shuffles matched or beat 0.115, so it failed a 5 percent threshold [3]. There is nothing conservative in that. As the author puts it, p=0.55 does not mean "maybe", it means a 55 percent chance the result is noise [16].
The number governing all of this is the discordant-pair count. The post reports a sign-test floor of roughly 0.031 one-sided for 5 discordant pairs out of 26 tasks [4]. That is 0.5 to the fifth power [1], and it is set by the five tasks that changed, not by the twenty-six that ran. Double it for a two-sided test and you get 0.062 [2], which is on the wrong side of 0.05, so by that arithmetic even a flawless edit fixing all five and breaking none does not clear the bar. The post describes the same case as barely clearing it [4].
Expanding the corpus made the arithmetic worse, because the edits did not scale with it. Mistral 24B, the strongest analyzer tried, returned positive deltas in three of five iterations at a mean delta of +0.025 and p around 0.5 [7]. On 40 tasks that mean delta is one task [4]. Holding the v0.1.0 effect size at 40 tasks would have required netting about five [3]. Two versions in, the most durable artifact the loop produced was a correct no.
Whether any of these p-values say something about your loop depends on your scorer. ExactMatch punished the v0.1.0 edit for adding "urgent" to tasks where the trigger keywords appeared but the label did not belong [10]. v0.2.0 added label-set-aware scorers, so "urgent, security" stopped failing against an expected "security, urgent" [11], and the author still concludes that any rule which adds labels struggles under strict scoring [12]. That is a property of multi-label triage with exact-set grading, and it holds discordant pairs per edit to a low count. If your tasks are more sensitive to prompt wording, or your scorer gives partial credit, a single edit touches more tasks and the same gate clears at the same corpus size. So the figure worth publishing is discordant pairs per edit; the p-value follows from it.
The author's diagnosis is that search is the bottleneck: the analyzer kept tightening urgency rules, over-corrected on others, never found a purely additive edit, and has no exploration mechanism [13]. Two releases of pipeline work were needed to reach that sentence [17]. The check that would have reached it sooner is a positive control, handing the gate an oracle edit that fixes every failure and seeing whether it still rejects, which the author says was never run and would have saved months [15].
Ranked by verification strength, evidence, and original report placement.
Mistral 24B, described as the strongest analyzer, produced positive deltas in 3 of 5 iterations with a mean delta of +0.025 at p around 0.5, roughly a coin-flip chance of noise.
Mistral moved roughly 1 to 2 tasks, directionally and inconsistently, and the author describes a +0.025 mean delta on 40 tasks as about one additional correct task.
In v0.1.0 of the self-improving prompt project, an analyzer edit fixed 4 tasks and broke 1 on a 26-task A/B corpus, a net of +3 with a mean delta of 0.115 and p=0.23.
The permutation test shuffles task labels 1,000 times and counts how often random chance produces a delta at least as large as the observed 0.115; it did so 23 percent of the time, above the 5 percent threshold.
The post reports the sign-test floor with 5 discordant pairs out of 26 as about 0.031 one-sided, and states that even a flawless edit fixing all 5 would barely clear p<0.05 two-sided.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · September 2, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
A self-editing prompt agent promoted noise because its gate tested p < 0.951 distinct publisher
build
One argument in a fix commit put the audited receipt in charge of its own verdict1 distinct publisher
product
Half the incident clock goes to search, and telemetry tools cannot read the answer1 distinct publisher
security
Thousands of credentials survived five years of pentests inside Jira ticket comments1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Checkable arithmetic, unreplicated experiment
Unusually for a self-published project post, the numbers survive being redone by hand: the 0.031 sign-test floor is precisely the one-in-32 that five same-direction discordant pairs give, and +0.025 across 40 tasks is exactly one extra correct task. What does not survive is scale — five analyzer iterations, one urgency-labelling task family, no one outside the project running the gate, and the author's own concession that the oracle positive control was skipped, which leaves 'weak analyzer' and 'flawed harness' still tangled.
One repo, one author, no outside users
The entire footprint is a tagged v0.2.0 release, a field test report and two source files in a personal GitHub repo. Nobody else is reported to have run this gate, on this corpus or any other, and the loop's own output was zero promoted edits across every model tested — so there is not even internal uptake to point at.
Honest headline, overreaching diagnosis
The framing concedes the result — the statistics killed the promotion, and the post leads with that — which is the opposite of overselling. The stretch is narrower and easier to miss: 'v0.2.0 proved the pipeline is correct' and 'we now know the exact mathematical relationship' are heavy words for five iterations of one model on one task family, in a post that also admits the confirming test was skipped. The p-value reading is likewise asserted rather than argued.
Author writing up his own repo, but selling a failure
This is the latest instalment in a series about the author's own project, closing with links to the release tag and the gate source — an obvious interest in the work mattering. The counterweight is real: the deliverable is a negative result, and someone inflating his own loop does not headline it with the statistics killing the promotion. Discount the diagnosis and the 'proved' language more than the measurements.
Single account, arithmetic holds, interpretation does not
One publisher, one author, one project he owns caps how far this can be trusted, whatever its internal quality. It sits above the floor because the quantitative claims are specific enough to check and they check out; it stays below the middle because the conclusions — pipeline sound, corpus adequate, search at fault — are inferences from five iterations, and the same run's significance is reported as both ~0.5 and 0.55.