Isolating every concurrent A/B test doubles your calendar for no statistical gain

A guide published on dev.to restates Optimizely's own position on concurrent experiments: overlap is the platform default, one visitor can sit in test A and test B at the same time, and mutual exclusion should be switched on only where it is genuinely required ^[1][2]. The reason to care is arithmetic rather than taste, because two tests placed in a 50/50 exclusion group each receive half your traffic and each takes roughly twice as long to reach significance ^[3].

The case for tolerating overlap rests on how assignment works. Optimizely hashes each experiment independently, so knowing which variation of A a visitor received tells you nothing about which variation of B they will get ^[4]. Visitors from A's control and A's variation therefore flow into B's control and B's variation proportionally, and whatever A does to behaviour is spread evenly across B's arms; per the guide, this adds noise but does not systematically favour one of B's variations ^[5]. Noise you can pay for with sample size. Bias you cannot.

The exception is an interaction effect, defined in the guide as exposure to one experiment changing how a user responds to another in a way that does not average out ^[6]. Its example is two treatments that are each defensible and incoherent in combination: test A rewrites the homepage hero to emphasise a discount, test B redesigns checkout and removes the promo-code field, and the visitor who was promised a code and finds nowhere to enter it abandons at an elevated rate ^[7]. B's results then report that the redesign loses, when it only loses alongside A's variation, and neither team can see that from their own results page ^[7].

Distortion scales with collision. According to the guide, two tests on unrelated pages measured on unrelated metrics essentially cannot produce the pattern, while two tests mutating the same funnel step almost cannot avoid it ^[8]. That is what makes a pre-launch screen cheap: ask whether the pair shares a page, a funnel, or a primary metric, and if so whether both touch the same element and the same hypothesis area ^[9].

Only after that screen does the decision have four options rather than two. Run concurrently at full traffic, which the guide calls the default and the fastest combined learning ^[10]. Combine into one experiment with more variations when both ideas target the same element toward the same goal, which measures the combination explicitly instead of leaving it to chance ^[11]. Run sequentially for zero interaction risk and zero configuration, paid for in calendar time ^[12]. Or use an exclusion group, where both tests run simultaneously and no visitor sees both ^[13].

Blanket isolation collapses that menu to its most expensive entry. Applied to a pair the screen would have cleared, an exclusion group buys a bias reduction of zero, since independent hashing already distributed A proportionally across B's arms, in exchange for roughly a 2x increase in run time ^[14].

One mechanic to hold onto before standardising anything: in Web Experimentation, exclusion groups are evaluated before page activation and audience targeting, with a visitor deterministically assigned to a group slot the moment they touch the snippet ^[15].

Work out what that evaluation order does to narrowly targeted tests sharing a group, since slot assignment happens ahead of the audience check ^[15]. Then audit last quarter's concurrent pairs against the three screening conditions ^[9] and count how many isolations you paid double run time for ^[3].