Loading today’s stories
Build1 distinct publisher2 min readPublished
A reader's rule said to suspect the test set before the methods, and checking it showed the fine-tune was the arm the easy data flattered most, losing 33 points on the rebuilt set against prompting's 28.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Follow any of these and your For You feed starts watching them — no settings page required.
build
A prompt A/B on six inputs is a coin flip until you have measured the noise floor1 distinct publisher
build
A floor pooled from seven control readings climbed above the effect it was measuring1 distinct publisher
build
The weighting ran, then the dedup pass undid it 56 lines later1 distinct publisher
build
Cheap GPU-hours, expensive finished jobs: what the comparison page leaves off1 distinct publisher
The v1 adapter scored 100 on the set it was generated from and 67 on the rebuilt one [1][11], and the second number came off the same weights and the same code [10]. Retraining that recipe on representative data pulled the 67 back to 95 [11]. So the 5 points the original table called a drop were 33 points of set difficulty with 28 points of retraining credited against them inside one cell [2]. Both numbers in the row were correct on their own terms, but the subtraction between them combined two different measurements, not one meaningful gap [8].
Once every row is one artifact measured twice, the losses order themselves by exposure to the v1 distribution: 33 points for the adapter trained on it, 28 for the prompted arm carrying six v1-shaped examples, 22 for the same prompt with no examples at all [12]. The 22 is the floor, the cost the harder set imposes on any model [13]. The 6 points between 22 and 28 are the only part the examples can be charged with [4][13].
That 6 is where the reader's kinship mechanism should have been visible [4]. Three of the six hand-written examples instantiate v1 generator templates, and git history shows they were not touched between the two evaluations [14]. The check was a matched replacement pool: six examples lifted from data/train.jsonl, the same 3 positive and 3 negative balance, the same three PII types, each verified absent from the test set by exact string match [15]. McNemar's exact test on the 400 paired predictions found 21 items only the old pool got right against 27 only the new pool got right, a net of six items across the whole set, at p = 0.47 [16][3]. According to the author, matching the pool's provenance to the test set produced a null result [17].
Treat the 33 as a statement about one bench: one base model, one adapter recipe, one PII labelling task, 400 evaluation items [15][9]. It transfers to your own comparison only if a checkpoint from the old regime still exists and the set is the single thing you change between runs. That is what kept this cheap, a single evaluate.py invocation against a retained adapter directory [9]. Keeping the superseded adapter on disk turned out to be the best-value instrumentation in the project.
The surviving claim is narrow. It concerns bookkeeping, not LoRA. A comparison row earns its subtraction when one artifact is held fixed and the test set is the only moving part [10].
Ranked by verification strength, evidence, and original report placement.
The author published a LoRA fine-tuning run two weeks earlier in which, on a self-generated v1 test set, the fine-tune scored 100% and few-shot prompting scored 94%, making fine-tuning look pointless.
On a test set rebuilt from real public corpora, the fine-tune scored 95% and prompting fell to 66%.
A reader called Max Quimby commented that the fine-tune lost 5 points moving to the rebuilt set while prompting lost 28, and that the distance between those drops says the original set was differentially easy for the prompted model rather than uniformly easy for both.
Quimby named a mechanism: the few-shot examples are part of the prompted method's input, so any kinship between the example pool and the test set inflates one arm and not the other.
Quimby closed with a rule: when two methods move by very different amounts after a test-set swap, suspect the set before you suspect the methods.
The author reports that Quimby was right the published table was misleading, wrong about which arm was inflated, and that the direct test of his mechanism produced a third answer neither of them predicted.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 31, 2026
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Fully shown, wholly unverified
The rescoring is about as legible as self-reported work gets: the command line is printed, the adapter path and 400-item limit are there, git history is invoked to prove the example pool never moved, and the two null findings are settled with paired McNemar tests rather than eyeballed margins. What holds the score down is that all of it lives in one person's repository on one dev.to post - no seeds, no repeated runs, no statement of base model or task beyond three PII types, and nobody outside the author has re-run a line of it.
No uptake to measure
Nothing here is a thing anyone adopts. The story is one developer re-running an evaluation on his own data with his own scripts; there are no users, no deployments, no downloads, no other practitioners reported as applying the same-artifact discipline. The three benchmark runs we can see are measurements, not evidence that anybody beyond the author has picked this up.
Undersold by its own headline
The framing is a mea culpa about a reader out-reading the author, which undersells what the afternoon actually produced. The piece could have stopped at the flattering half of the reader's thesis; instead it decomposes a 28-point loss into a 22-point set penalty plus 6 points of leakage, then kills the leakage remedy with p = 0.47 and says so. The correction also runs against the author's earlier headline rather than for it. The reason this is only mildly negative rather than strongly so is scope: 400 items, one task, one model, one run each, and the write-up does not caution readers about that.
Author grading his own homework
One writer, correcting one post of his own, using numbers only he can produce - the setup invites tidying, and a public retraction that ends in a clean monotonic ordering is itself good copy. Against that: the correction turns his own 5-point figure into 33, concedes the mistake was his fault and 'not the reader's', and reports a null against the mechanism he set out to confirm. Real pressure, visibly resisted, with no sponsor, vendor or product anywhere in the frame.
Coherent, single-witnessed
The internal arithmetic checks out - 33 minus the 28 that retraining recovered leaves the 5 that was published, 22 plus 6 makes prompting's 28 - and the dates and commands are specific enough to argue with. But a story this self-contained is only as good as its one witness, and a single unrepeated run per cell means a couple of these gaps sit close to the noise the piece never quantifies.