Microsoft's experimentation team tells experimenters to relax about overlapping A/B tests

A scan that size produces alarming-looking pairs on its own. Under the null hypothesis of no interaction the p-values are uniform, so 5 percent fall below 0.05 and 0.1 percent below 0.001 ^[13]. Take 300,000, a round figure inside "hundreds of thousands" ^[12]: chance alone puts about 15,000 pair-metric combinations under 0.05 and about 300 under 0.001 ^[17]. Any finding here has to be a statement about the shape of the whole distribution rather than a count of significant pairs. The post's text ends after it sets out that null expectation, before the observed counts ^[16].

The cost of caution the team names is statistical power. When it suspects two tests will interact, it isolates them so users are not assigned independently for each test, and that reduces the users available to each one ^[6]. Suppose isolation puts two tests on disjoint halves of the traffic. Each sample halves, and because the smallest detectable effect scales with one over the square root of the sample, the smallest effect each test can resolve grows by roughly 41 percent ^[18]. A team that could have read a 1 percent movement now needs 1.4 percent.

The team was explicit about the limits of its own advice. Interactions are "rare enough that you can usually run your tests without worrying about them," the post said ^[14], while adding that "we're not saying that you can completely let down your guard and ignore A/B interactions altogether" ^[15]. Microsoft's earlier experience pointed the same way, and researchers at Meta found interactions were not a serious problem for their tests ^[7].

The design is narrow in two ways. Each of the four products contributed a single day, and all four sit inside one product group where assignment is already independent ^[9][8]. A team whose concurrent tests all touch the same surface, as in the post's own case of ad text turning red on a background turning red ^[5], is not the portfolio being measured. The post says isolation remains in use for pairs the team suspects in advance ^[6].

Sequencing matters for anyone copying the default. Independent assignment is cheap to switch on and expensive to audit afterwards: establishing whether overlapping tests distorted last quarter's scorecards takes the computation the platform team ran, which is every product metric under every control and treatment combination for every pair running that day ^[10]. A team that relaxes the default without keeping that computation available has no way to size the damage later.