A fanless MacBook turned a thread-scaling sweep into a 54% error, and averaging would not fix it

A developer benchmarking how a search engine scales with thread count on a fanless MacBook Air (Apple M4) discarded an entire first data set after a sanity re-run under the same conditions returned figures up to 54 percent different ^[1][2][3]. That matters well beyond one laptop, because on a passively cooled machine the order in which you measure configurations leaks into the numbers as a systematic bias, and the standard defence of running the sweep again and averaging preserves it ^[4][5].

The original protocol measured 1 through 10 threads in order, and the curve looked clean enough to carry a headline result: 11 threads at 5.2 seconds, the fastest run in the set ^[1][6]. On the re-run, that configuration came back at 8.0 seconds, an increase of about 54 percent ^[7][1]. The 4-thread case moved the other way, from 11.0s to 9.0s, roughly 18 percent faster ^[8][2]. The author reports no rogue background process and attributes the spread to the chassis: under sustained load a fanless machine heats up and thermal-throttles, so every measurement depends on what was measured immediately before it ^[9][3][10].

The useful part of the write-up is the distinction it draws. Sweeping ascending means the high thread counts always execute on a hotter chip, so the penalty lands consistently in one direction, and repeating the whole biased sweep and averaging reproduces the bias rather than cancelling it ^[4][5]. The 5.2s record, on the author's account, was a single lucky run on a cold chip ^[11]. His summary is worth keeping: "The cleaner the graph, the more suspicious you should be" ^[12].

The replacement protocol sweeps 1 to 16 threads once, repeats the entire sweep for 4 rounds, and takes the per-thread-count median across rounds ^[13]. That produced 128 measurements and, per the author, a plateau at 8 threads at 3.0x, with the 5.2s figure gone and 7.4s standing as a reproducible floor ^[14]. Sixteen thread counts across four rounds is 64 runs, so 128 is consistent with two independent series ^[3], which fits the validation he offers: two search patterns with hit counts an order of magnitude apart, 10,967 versus 722, yielded nearly identical scaling curves ^[15]. That is the right kind of check. Re-running a biased protocol only confirms the bias; two unrelated workloads agreeing is evidence the thermal signal is no longer riding along ^[5][15].

One caveat the post does not address. Fixed-order round-robin shrinks the gradient but does not eliminate it: in every round, 16 threads still runs after 1 through 15, so the last configurations still see more accumulated heat, just 4 rounds' worth of it instead of 64 runs' worth. Shuffling the order within each round is what actually decorrelates position from configuration and converts residual thermal state into noise a median can absorb. The post also has internal seams: the first sweep is described as 1 through 10 in order, yet the record sits at 11 threads, and the corrected sweep runs to 16 ^[1][6][13]. The publisher's own note says accuracy and completeness are not guaranteed ^[16].

Watch whether the 8-thread plateau survives being measured with randomised within-round ordering and logged idle gaps, and whether benchmark method sections start reporting run order at all. The author kept the discarded CSV but ruled out publishing it, on the grounds that a retracted number left lying around eventually gets cited ^[17].