Shopify's continual learning loop cuts serving costs 96%; case study separately details judges calibrated from 25 annotated samples

A 96 percent cut leaves serving at 4 percent of the old cost, a factor of 25 ^[1]. Shopify's figure is for one agent: the GraphQL agent that the PyTorch case study names as its clearest example of the loop running in production ^[2].

The reason to build the loop at all is in the weights. A deployed frontier model's weights are frozen, so improvements pile up around the model instead, in prompt edits, retrieval examples, routing rules and harness code ^[4].

Quality definition is the step the case study says teams rush. The specification becomes the reward signal, and when the evals or specs are wrong, everything downstream optimizes the wrong behaviour ^[8]. Annotators do not get to hand in a score and one sentence, because the calibration algorithms need the reasoning behind every score to learn from ^[12].

The gate comes before any training. Two of the best annotators or product experts blindly score 25 random samples, and their inter-annotator agreement is recorded as Cohen's kappa, which measures agreement above chance ^[9]. Fifty judgements in total ^[2]. If kappa lands around 0.2, the case study calls the rubric ambiguous and says to meet again and iterate, on the grounds that a rubric which confuses several product experts who work on the product every day will confuse an LLM too ^[10]. That agreement is also the judge's ceiling, and the stated goal is a judge that matches humans about as well as humans match each other ^[11].

Calibration then turns the rubric into something that can be run on unlimited production datapoints ^[13]. Shopify uses DSPy with reflection-based optimizers: GEPA, which evolves the prompt by reflecting on natural-language failure traces and keeps a Pareto frontier of candidates instead of greedily choosing a single winner, and Agentic Context Engineering, which curates a structured playbook through small incremental edits ^[14]. Because the judge is an offline proxy, they backtest it against previous A/B tests and ask whether it recovers the direction of known wins and losses in engagement and retention ^[15].

For the 96 percent to mean anything elsewhere, several conditions have to hold. Request volume has to be high enough that serving dominates the bill, which is the problem the case study describes when frontier models get too slow and expensive to serve every request at scale ^[5]. The scoring criteria have to be cheap enough for a calibrated judge to apply to a full day of traffic. The spec has to be stable enough that last quarter's ground truth still describes good behaviour. And there has to be a set of past A/B tests with known outcomes; a team shipping its first AI feature does not have one ^[15].

The case study reports 96 percent as an outcome, and its sections on defining quality and calibrating the judge do not give the serving baseline, the base model, or what a day of training costs ^[16]. In my view this is the right trade for a high-volume agent against a fixed schema. For a product still changing shape every week, the rubric would be stale before the judge was calibrated.