Spotify traces the default Bayesian A/B stopping rule back to peeking

Set the stopping rule to a Bayes factor threshold and you get false positive control even under repeated looks, because the Bayes factor is a martingale under the null ^[6]. A calibrated empirical Bayes prior instead shrinks effect estimates to counter winner's curse bias and bounds the false discovery rate ^[7]. Decision-theoretic formulations derive stopping rules that minimise a chosen cost function, and sometimes control frequentist rates, sometimes not ^[8]. Four configurations carry four different sets of guarantees. By the post's own account, the one shipped as the default is the only one of the four that reproduces peeking ^[5][17]. The framework, as Spotify describes it, spans state-of-the-art sequential methods and procedures numerically identical to bad frequentist practice ^[9].

That is the substance behind the interpretability pitch. The argument Spotify quotes is that Bayes is modern, flexible and easier to interpret, and that frequentist complications like multiple testing and sequential testing fall out naturally in the Bayesian way of thinking ^[2]. Spotify's answer is that this is a theme of oversimplification that leads to inference no better than peeking ^[3]. Easier to interpret describes the sentence on the dashboard. The false positive rate is set by the stopping rule, not by that description.

The load-bearing argument is smaller than the Bayes-versus-frequentist framing suggests. False positives and false negatives are a classification of outcomes that any experiment program accumulates over time whatever philosophy its inference rests on, so the question is whether controlling those rates is a goal and whether the chosen configuration has parameters that control them ^[12]. Spotify puts goals first: limiting the number of shipped features with no effect, estimating impact to a stated precision, or minimising a cost function over time, with the statistical configuration following from those goals and constraints ^[13]. That ordering, reversed, produces the confusion the post describes, where a fact about one goal paired with one configuration is repeated as a fact about Bayesian A/B testing ^[15].

The taxonomy carries over; the verdict does not. Spotify's stated conclusion is about its own program being served by tooling it already runs ^[11]. If your program's goal is a cost function minimised over time, the branch that addresses it is the decision-theoretic one ^[8][13], and the post also argues those Bayesian formulations can be translated into equivalent frequentist formulations ^[14]. The live choice is which parameters you can set.

The check is cheap: a platform's Bayesian documentation states two values, the default prior and the stopping rule. Flat prior, posterior-probability threshold, dashboard refreshed every morning, and that is the configuration the post says reproduces the error rate of peeking ^[5]. A Bayes factor threshold or a calibrated prior is a different procedure with different guarantees, sold under the same word ^[6][7].