Raising the seed count from three to ten dropped most RL policies below random

A seed is supposed to be the only thing that varies. Pin it, and the same script should walk the same fifteen steps and return the same hit count on any machine. The author of the dev.to post got the opposite. Hit counts, statistical significance and the ranking of the policies all changed when he moved the experiment to different hosts. He could not pinpoint where the differences came from ^[7][8]. The script had no CUDA-like code, so the GPUs and TPUs on those hosts were not in the loop, and the moderate speedup he measured came from faster CPUs ^[6].

A ranking that moves with the host means the run is taking an input the seed does not cover. He read it as a stochastic process with a stable conclusion underneath. He carried on ^[9].

The sample size is the easiest part of this to check. Five policies at three seeds each is fifteen runs, and those fifteen runs are the entire basis of the clean result ^[1]. Ten seeds per policy is fifty ^[1]. At fifty, many of the policies trailed Random, and by his own account it may come down to one policy still showing a significant improvement over random modification ^[13]. Random was one of the five and also the baseline, so four policies were learned, and at most one of those four survived better sampling ^[3].

The interpretable weights were the part that made the result feel mechanistic, and the mechanism was one he had already doubted. A single matrix retaining functional-group preferences across chemically different starting compounds looked naive to him, because different scaffolds need different modifications ^[10]. He kept it. He wrote that he kept telling himself "Yet it worked" and "The stats don't lie" ^[11].

What the post does not supply is the defect itself. He writes that the catastrophe was not RL failing, since he already had doubts that his setup was enough for the agents to learn properly, and that the real issue was something else ^[14]. The supplied text breaks off mid-sentence at exactly that point ^[15]. The harness reading therefore rests on the sequence he does document: host-dependent rankings first, collapse under more seeds second.

Runtime is what capped the seeds. The script was slow, so three per policy was the minimum he was comfortable with. He raised it to ten when he thought he was ready to share the results ^[12]. That is 35 more runs, about 3.3 times the original count, to establish that the first answer was luck ^[2].