Here is what actually happens when you call the Qiskit gym. `generate_preset_pass_manager` takes a `seed_transpiler` argument that pins the routing pass, which uses randomness [6]. Benchpress does not pass it [2]. Circuit construction is pinned, `seed=12345`, fourteen times over [7]. The compilation is not. So every gate count the suite prints is one draw from a distribution nobody measured [7].
The BQSKit gym in the same repository does seed its compiler, at `seed=0` [8]. `seed_transpiler` is a Qiskit-specific API the other gyms cannot call, so this is a Qiskit-gym problem and only that [8]. That is worth stating plainly, because it narrows the fix to one line rather than a rewrite.
The consequence is measurable. On `bv_n140`, a Bernstein-Vazirani circuit named in Qiskit issue #14402, mapped to a heavy-hex lattice from Qiskit 1.4.3 to 2.0.0, the true change over 400 seeds per version is +5.37% [3]. Under the suite's own three-run protocol the verdict crosses the 10% regression line 24.4% of the time [1]. The compiler got a little slower, and the instrument calls it a regression a quarter of the time anyway.
"Run it more" does not rescue you cheaply. Twenty runs per version is roughly forty hours of compute at the issue's stated two hours per suite run, and it still leaves a 3.74% error rate [4]. Taking the minimum of three runs instead of the mean, a common instinct, roughly doubles every rate: `bv_n280` goes from 17.3% to 28.4% [5]. The minimum is the wrong estimator when the noise is the thing you are measuring.
The author checked the number he trusted least before publishing it. The original 200 seeds all came from one contiguous block inside one process, holding hash randomisation, allocator layout, and any cached RNG constant [9]. A disjoint set of 200 seeds spanning 5.9 million to 1.08 billion, run across ten fresh processes with differing `PYTHONHASHSEED`, returned 22.6% against the contiguous set's 26.3%, each inside the other's interval [9]. The variance is real; it is not an artefact of how the seeds were laid out.
That reframes issue #14402 itself. It reports +46.1% for `bv_n280` [10]. A single three-run comparison of the same circuit under the same protocol returns anywhere from -10.5% to +100.0%, with a 95% range of [+9.3%, +57.2%] [10]. The +46.1% sits at the 88th percentile of that distribution; the same range extends below the 10% threshold [11]. The reported figure is real: it is one draw reported as a measurement.
To keep one circuit from being an anecdote, the analysis was pre-registered: selection rule, endpoint, and labels committed before the data, with checkable commit timestamps [12]. Across the eligible set, 12 of 26 circuits have a decision-error interval that excludes zero, 46.2% with a Wilson 95% of [28.8%, 64.5%] [13].
Treat that 46.2% carefully. The 26 circuits come from eleven algorithm families, and within a family results are nearly all-hit or all-miss [14]. A cluster bootstrap over families widens the interval to [17.4%, 81.0%] [14]. The effect is not 26 independent observations; the honest sample is closer to eleven. The direction survives that correction, though the precision does not.
