The in-fold failure in the dev.to write-up deserves more attention than the leak itself, because it is not an artefact you can subtract out afterwards. The meta-learner did what it was built to do, which is regress the target on member predictions and keep whichever column explains most. When one member is a bagged forest scoring rows it helped memorise, that column looks like an oracle, so it collects a weight of 1.123 while the gradient booster that was actually better gets 0.035 [2]. That is roughly 32 to 1 in favour of the worse model [3]. Throw away the inflated score and the wrong ensemble is still the one you shipped [14].
Now the arithmetic on the honest version. Out-of-fold stacking's advantage over the best single model is 0.0030 [1]. The set publishes no seed spread for that run at all; the only spreads in it belong to the search posts, at 0.0098, 0.0128 and 0.0004 [8]. The smaller of the two large ones is a bit over three times the whole stacking gain [2]. Different metric, likely a different dataset, so read that ratio as indicative rather than exact. It is still the only noise estimate anywhere near the comparison, and it is larger than the effect being reported.
The tuning half needs no such inference. The measured grid-versus-random gaps were 0.0008 one way on three parameters and 0.0039 the other way on six [4], both inside the 0.0098 band [3], and grid won on exactly five of ten seeds, which makes a single-seed run of that experiment a coin toss on which headline you get to publish [5].
Two results survive their own variance. The one method whose advantage grew with budget went 0.0032, then 0.0051, then 0.0063 at 20, 50 and 120 trials [6], roughly doubling for six times the trials [8], and monotone movement in the direction the mechanism predicts is harder to get out of noise. The author's own thirty-line search landed within 0.0004 across three seeds where random search spanned 0.0128, for a mean advantage of just 0.0015 [7]. That sells stability, not accuracy.
Blending is the same lesson with a larger number on it: on the thin holdout it reported itself 0.1084 better than it delivered [5] and finished 0.0061 behind an unweighted average that cost nothing [4]. The pruning entry prices the brochure. Killing 12 of 20 trials is a 60 percent kill rate against a 28 percent wall-clock saving [6], because a killed trial still pays for its warm-up [9], and the log loss for that trade was 0.0005 [7].
The same habit shows up where the author turns it on his own distribution. Four of the six posts went out inside ten minutes, and the first two took 20 and 12 views while the last two took 1 and 0 [12], which is 32 against 1 [9]. Six posts, zero substantive comments [12]. The rule he ends on is the transferable part: run it twice before you believe it, and check that what you are comparing is bigger than what you are not controlling [13].
