Repeated A/A tests turned up a 30% false-positive rate on one experimentation platform

Start with the chi-squared, since it checks by hand. An intended 50/50 on 100,000 visitors puts 50,000 in each arm. The observed 52,000 against 48,000 misses that by 2,000 on each side. Square 2,000, divide by 50,000, and each cell contributes 80; two cells give 160 ^[9][18]. The post puts the matching p at about 10 to the minus 36 ^[9].

Teams flag sample ratio mismatch at p < 0.001 instead of 0.05 for exactly this reason ^[8]. At production volume a trivial imbalance clears 0.05 without effort, and the tighter bar trades a few false alarms for not missing a broken splitter ^[8].

What breaks a split sits upstream of the statistics: bot traffic skewed into one arm, an ID collision, a redirect that drops users out of their assigned bucket before they are logged ^[10]. The metric computation runs the same way through all of them. So the p-value comes out correct on a comparison the split has already corrupted ^[12].

The A/A test aims at exactly this class of fault. According to the post, Microsoft's experimentation team documented a platform where repeated A/A runs turned up a false-positive rate as high as 30% against a nominal 5%, with the stats engine itself miscalibrated ^[3]. That is six times the design rate, about 30 of every 100 null comparisons flagging a difference where 5 was intended ^[17][20]. The arithmetic downstream of a miscalibrated engine can be entirely correct ^[4]. The post does not name the platform or cite the write-up ^[21].

So the 30% is a measurement of someone else's engine. For the number to transfer, your platform would need the same class of miscalibration, and the way to learn your own figure is to run two identical arms against each other on your own pipeline ^[1]. If it reports a winner, the fault is in the testing setup ^[2].

The continuous version is a platform change. Traffic that is not allocated to a real test goes into a permanent A/A, so a regression in the pipeline surfaces before it corrupts an experiment someone is reading ^[5]. The post calls A/A the cheapest validity check available, underused because it feels like it cannot fail ^[16]. Routing the spare traffic needs an allocation layer willing to assign users to a null test, plus something watching the output.

Simpson's paradox is the case where the aggregate and the segment numbers are both correct and tell opposite stories ^[22]. In the 1986 kidney stone study of 700 patients, Treatment B beat Treatment A overall at 83% against 78%, and A won in both the small-stone and the large-stone group ^[13]. Doctors had given the gentler Treatment B to patients with smaller stones, who were easier cases whatever the treatment ^[14]. Nobody split anything wrong there; assignment was clinical judgement ^[15]. In a randomized test the same reversal means something correlated with the outcome landed unevenly across the arms, and the post treats that as the stronger signal of the two ^[15]. The five-point aggregate gap and both subgroup wins are arithmetically correct ^[19].