Build1 publisher3 min readPublished
Five commits in the fixture left a ranking test comparing string lengths
For its whole life the assertion turned on the 41 characters of an Applies to: header, and it took a Windows run where the string lengths fell the other way to expose that. Two more checks in the same suite were green for their own separate reasons.
The Engineer · Build desk

What happened
- A second check forced sys.platform to win32, exercised a directory-search detector, restored the platform and then asserted silence, so it failed on Windows when the detector correctly fired.
- A CI timing check reported a redaction routine at 0.0 ms for 4 KiB of input and 31.2 ms for 16 KiB, printing that as a 625-fold growth for four times the data.
- All three checks had been green for reasons unrelated to what they claimed to test, and the author found them only because CI ran the suite on a platform they do not own.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A green run on one machine cannot separate an assertion that held from an assertion that was never reached, so the second operating system in CI was the only auditor these three checks had.
- decision Any fixture that has to exceed a threshold in production code should read that threshold from the code, because a hardcoded count silently stops crossing it the moment the constant moves.
- exposure The scoped-rule feature went into the release notes on the strength of a check that had never exercised it, so users got the claim while the suite held no evidence for it.
- precedent A ratio with a floored divisor will keep producing catastrophic-looking growth factors on any runner whose clock is coarser than the fast case, and the failure will be read as a code regression.
The x625.0 in that CI log is a claim about the Windows scheduler, not about the redaction routine [15]. The line doing the dividing is `ratio = big / max(small, 0.05)` [16]. On Windows, `process_time()` advances on the scheduler tick, roughly every 15.6 ms, so the 4 KiB case finishes inside a tick and reads 0.0, the floor supplies 0.05, and 31.2 divided by 0.05 is 624 [17]. The log printed x625.0 [15]. Both measurements were quantised to ticks: 31.2 is exactly 15.6 doubled [18]. The same input passed on Python 3.8 in the same run, and a genuine quadratic blowup does not care which interpreter runs it [19].
The second attempt is the more useful failure. It keyed the autorange target off `time.get_clock_info("process_time").resolution`, which returns 1e-07 on Windows, the unit `GetProcessTimes` reports values in and not the rate at which those values change [21]. The target would have been 0.002 ms, so the autorange would have stopped on the first iteration and the fix would have been a no-op on the only platform it exists for [22]. It would also have passed every local run, because the author's Mac really does resolve to a microsecond and the repeat count stays at 1 there [23]. The version that shipped spins on `process_time()` until the value changes, then takes the larger of the measured tick and the reported resolution; on Windows the target becomes about 312 ms [24]. That target is twenty ticks [4], and 156,000 times what the documented API would have produced [5].
The ranking check failed for a duller reason. Its assertion compared the raw length of two strings, and the scoped rule's body wins because it carries the Applies to: line, 41 characters [6]. The check had been passing on the length of the line that declares the scope it was supposed to be testing. Its fixture committed five times into a temp repository, a tenth of the fifty commits the ranking needs before it returns anything [5][3]. "Windows didn't find a Windows bug. Windows got unlucky with string lengths in the other direction and exposed a check that had never worked," the author wrote [7]. The fixture now derives its commit count from the constant, `range(_rollup.MIN_COMMITS_TO_RANK + 2)` [8]. With the trigger firing, the three context blocks measure 541, 306 and 300 characters, against 411, 370 and 371 before [9]: a real margin of 235 characters where the accidental one was 41 [1][2].
The middle case is the one I would expect to find in most suites. The check sets `sys.platform` to "win32", exercises the detector, restores the platform, and only then asserts that the detector stays quiet [12]. On Windows the real platform is the platform under test, the detector correctly fires, and the check goes red [13]. "I'd written a check whose final assertion was only correct on machines that are not the thing it tests," the author wrote [14].
For any of this to transfer, a suite needs one of three shapes: a fixture whose magnitude is compared against a constant that lives in the code under test [5], an assertion that runs outside the region it patched [12], or a ratio with a floored divisor and a clock coarser than its fast case [16][17]. The author does not report running mutation testing or a fixture audit. All three surfaced because CI ran the suite on a platform the author does not own [2]. I would expect the first two to show up as surviving mutants, since one never reaches the ranking code and the other asserts a negative. The timing check is a defect in the harness's own measurement, and no mutant of the redaction routine changes what a floored divisor prints [16].
What to watch
- Whether the autorange holds on Windows runners whose scheduler tick is not 15.6 ms.
- Whether the suite adds an assertion that the ranking trigger fired at all, not only that the scoped rule won.
- Whether other timing checks in the same harness still divide by the 0.05 ms floor.