Spotify's Researchers on LLM A/B Tests: You Can Have Speed or Identification, Not Both

Spotify's engineering blog has published a post arguing that LLM predictions can stand in for human outcomes in an A/B test "only by assumption, not by design" ^[1]. That is a narrower claim than it sounds, and a more damaging one: randomized experiments are treated as the gold standard because randomization causally identifies the treatment effect by design ^[2], and substituting LLM-generated predictions for real user responses removes that guarantee, leaving identification to rest on assumptions instead ^[3].

The pitch for simulated experiments is familiar: run it on a model rather than on users, get results in hours instead of weeks, and skip traffic allocation entirely ^[4]. According to the authors, most proposals in this vein skip the statistical question that makes an experiment valid in the first place, which is under what conditions the experiment identifies the effect you care about ^[5]. Their paper formalizes those conditions using surrogate endpoint theory from biostatistics ^[6], the same framework that justifies using lab biomarkers as fast, cheap stand-ins for clinical outcomes in trials ^[7].

The empirical part is the useful part. The team used the Upworthy Research Archive, described in the post as the largest open-access A/B test dataset available, containing click-through rates for news headline variants across thousands of tests ^[8]. They prompted gpt-4o-mini to predict the click-through rate of a typical user for each headline, separately for treatment and control ^[9]. Dropping those raw predictions into a standard experimental analysis recovered 39 percent of the observed human treatment effect ^[10]. That leaves 61 percent of the measured effect unrecovered ^[11], and the authors are explicit that this is not noise you can average away: the bias is systematic and directional, attenuating effects toward zero so treatments look weaker than they are ^[12]. Run that across many features and product areas and the organization systematically underestimates the value it delivers to users, with shipping decisions to match ^[13].

Two conditions make the substitution legitimate. Surrogacy requires that the LLM output fully mediates the treatment effect on the human outcome, so that once you condition on the prediction and on baseline covariates, knowing which arm a user was assigned to tells you nothing more about their behaviour ^[14]. The post notes this is often implicitly assumed in LLM-based A/B testing but rarely spelled out and not commonly validated ^[15]. Comparability requires that the calibration function mapping LLM predictions to human outcomes is the same in the new experiment as in the historical data used to fit it ^[16]; strengthened to stability of the full joint distribution of pre-treatment characteristics and predictions, it buys you quantities beyond the average effect ^[17].

Calibrating LLM outputs against human data did recover the treatment effect on Upworthy, but only with a specific set of methods ^[18]. The trap is in the next sentence: these conditions cannot be verified for new treatments, and they get less plausible the further a new treatment sits from the experiments you already ran ^[19]. As the authors put it, the promise is least justified precisely when it offers the most benefit ^[20].

Watch for whether anyone selling simulated experimentation states which calibration estimator they use and on which historical corpus it was fit ^[18], and whether they hold back real traffic to test comparability rather than assuming it ^[16]. A vendor claim of unbiased effect recovery on headline-style copy tells you nothing about a new surface. Also worth noting: the published excerpt breaks off mid-sentence where the calibration methods are introduced, so the estimator comparison is not yet readable in the post ^[21].