Loading today’s stories
Build1 distinct publisher3 min readPublished
AgentSelfEdit's promotion gate compared the p-value against the confidence level instead of alpha, widening its acceptance window nineteenfold. Its author reports 31 fixes in one session, nine of which had been faking a working system.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Take the comparison line on its own terms. `passed = p < confidence_level`, with `confidence_level` set to 0.95, accepts every p-value below 0.95 [4]. The intended test accepts everything below alpha, which is 1 minus the confidence level, or 0.05 [14]. Under the null hypothesis a p-value is uniform, so the shipped line admits roughly 95% of pure noise where the design admitted 5%: an acceptance window 19 times wider [1]. The variable name is the trap. `confidence_level` reads like a threshold, and it is one, just not for p.
The second bug is the one I would also have shipped. In `run.py` the loop passed `proposal.new_text` as `prompt_b`, and `new_text` is the edited fragment, not a complete system prompt [10]. The A/B engine did not raise on the invalid input. It fell back to the current prompt for both arms [10]. Ten calls went out, five tasks times two arms, every one of them carrying identical prompt text [9], and the run printed `A/B test: tie (p=1.0000, n=5)` followed by `Gate: reject` [8]. The pipeline completed and the gate returned the correct verdict by an entirely fictional route, which is why nothing rang. The tell was zero variance across five tasks, and according to the dev.to writeup that is what sent the author into the traffic logs [9].
Hold that `n=5` next to the corrected threshold. If the gate's statistic were a one-sided sign test over five paired tasks, the smallest reachable p is 0.5^5, or 0.031, so only a clean five-nil sweep would clear 0.05 [4]. The reported post-fix value of p=0.23 [7] says the actual test is something else, but the coarseness stands: a correctly wired gate reading five held-out tasks rejects nearly everything. That value sits at 4.6 times the threshold it has to beat [2]. And the celebrated 20%-to-40% accuracy jump [6] is one task changing answer, if it was scored on that same five-task set [5].
What would have to be true for any of this to transfer. It is one author, one open-source sidecar that rewrites its own system prompt from execution feedback, self-reported, 31 issues fixed in a single session with nine of them changing the system's character rather than its behaviour [1][3] -- 29% of the total [3]. That is not a survey of agent harnesses. The transferable part is the other seven: scoring accepted any non-empty response, the CLI talked to a mock instead of a real LLM, the config silently ignored the endpoint, the Docker test skipped the hard parts, the failure traces were fabricated, the gate received the wrong prompt, and the field-test runner measured the wrong thing [12]. Each of those is a component that chose to emit output rather than refuse.
The audit that catches the whole class is dull. Feed the gate two arms you know are identical and assert reject. Feed it a candidate you know is better and assert accept. Then flip the comparison operator in a scratch branch and confirm the suite goes red. If inverting an operator does not break a test, the test was measuring completion.
Ranked by verification strength, evidence, and original report placement.
AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback, A/B tests the edits, and promotes only statistically-proven winners.
The project code is published at github.com/deghosal-2026/agent-self-edit.
The author reports finding and fixing 31 issues over a single session, nine of which were fundamental in that each made the system look like it was working when it was not.
The promotion gate was written as `passed = p < confidence_level`, i.e. p < 0.95, instead of `alpha = 1 - confidence_level; passed = p < alpha`, i.e. p < 0.05.
With the buggy gate, a p-value of 0.9 would pass and even 0.5 would pass; a promotion at p=0.1 carried a 10% chance of being random noise.
The gate promoted an edit whose reported accuracy jumped from 20% to 40%.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 31, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
A gate that stops firing shows up in PlannerCritic's metrics as safer plans1 distinct publisher
build
One argument in a fix commit put the audited receipt in charge of its own verdict1 distinct publisher
build
The .mp4 that was never H.264: how a healthy serving path hid a codec bug1 distinct publisher
build
The prompt never arrived: a Windows batch shim was worth 15 of 24 runs in an agent eval1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Falsifiable in principle, unchecked in practice
The strength here is granularity: an exact broken comparison, an exact one-line repair, a log line quoted character for character, a before-and-after p-value. The weakness is that all of it comes from the one person who wrote the code and no one has opened the repository to confirm any of it. Two of the sharper numbers — the sign-test floor and the claim that 20% to 40% is one task flipping — depend on assumptions about the gate's statistic and task set that the post never states.
A public URL is not a user
The only thing observable about uptake is that the code exists at a public address. No stars, forks, installs, dependent projects, or anyone other than the author running the loop appears in this reporting — and a session of bug fixes on one's own machine says nothing about whether anybody else has tried it.
Undersold, with one boast left standing
Most stories drift the other way. This one leads with 'it looked like it worked, it didn't' and spends its length itemising ways the author fooled himself, which if anything undersells a genuinely transferable lesson about gates that never reject. The single overstatement is inherited: the project still bills itself as promoting only statistically proven winners, a line the same post shows was false for as long as the gate compared p against 0.95.
Reputation is the only currency
Nothing is being sold. The author's payoff is credibility and eyes on a repository, and the chosen route to both is admitting nine ways his own system faked success — an incentive that pushes toward candour on the bugs while quietly leaving the project's marketing line intact. There is no vendor, funder, or benchmark result anywhere in this reporting to pull it the other way.
Internally consistent, externally untested
The arithmetic holds up wherever it can be checked — 0.95 against 0.05 really is a nineteenfold window, 0.23 really is 4.6 times 0.05, nine of 31 really is 29% — and the narrative never contradicts itself. What caps our confidence is structural: one publisher, one author, one machine, and two of the more quotable inferences resting on premises the post declines to state.