Nine plugins make 36 unordered pairs. A Kendall's tau of -0.50 across those pairs works out to 27 discordant and 9 concordant, assuming no ties, so three quarters of the head-to-head comparisons flip when the ranking axis changes from backup duration to archive size [18]. The two orders are not loosely related. They are mostly reversed [4].
The cause is arithmetic in the CPU rather than a quirk of one vendor: compression spends time to save bytes, so the plugin that skips it finishes sooner and ships a fatter archive [5]. A wall-clock ranking is therefore partly a ranking of who compressed least. Six of the nine plugins are beaten on neither axis, and only three are beaten on both [6]. Arguing that the top line of your own leaderboard should not exist is an unusual move from second place, which is where the benchmark's author sits, by his own disclosure, as the seller of one of the nine plugins measured [1].
For the -0.50 to transfer to a benchmark someone hands you, the two axes on that board would have to trade against each other by construction, the way CPU time trades for bytes here. Where the axes move together, one number is a fair summary. The portable part is the question rather than the coefficient: which second metric was measured, and what weighting collapsed it into the headline figure.
SPEC has had a written position on this for years. Its Fair Use rules define a derived value as any number that is a function of a benchmark metric plus something else, permit composites outright, and prohibit presenting one as the benchmark's own metric [7]. The worked example is a gaming society building a composite from weighted subsets of two SPEC benchmarks: useful and interesting, says SPEC, but the weighting and the subsetting were the society's work, so calling it SPECgame would be a violation [8]. Two neighbouring rules do the enforcement. The basis for comparison has to be stated, meaning fastest among what, measured how, retrieved when [9]. Several benchmarks also require named metrics to be quoted alongside any headline number, so the flattering figure cannot travel alone [10].
The error bar had the same disclosure problem in a different coat. The board started with CI = 1.96 * (400 / sqrt(n)) over battles played, which gives +/- 351 at five battles and +/- 55 at 200 [11], and which assumes every battle is an independent observation [12]. One record had 24 of its 40 battles fail on the same fault reproduced from a single bug [13]. That is one fact seen 24 times. The fix treats outcomes sharing a root cause as perfectly correlated, so a cluster of any size counts once, with failures grouped by harness version, hosting tier and phase [14][16]. The code subtracts size minus one per cluster and floors the result at the number of distinct clusters seen [15]. Forty minus 23 is 17 effective observations [20], and the interval moves from +/- 124 to +/- 190 [17], about 53 percent wider [19].
Wider is the direction the evidence pushes, since the assumption being replaced was the flattering one. Both corrections cost the author his headline: one number becomes two axes plus a stated weighting, and a tight interval becomes a loose one that survives contact with a single bug.
