Physicists' formula for when chatbots turn bad got 15 of 16 small-model tests right

Fifteen of 16 is 93.75%, rounded to 94% in Decrypt's account, and it is one miss ^[14]. The 16 were described as clear-cut cases. Each test was a binary call on whether a model would tip at once or after a run of good tokens ^[3]. The preprint's predictions could also be off by one output ^[9].

The six open-weight models in the preprint came from OpenAI, EleutherAI and Meta and ran from 124 million to 410 million parameters ^[8], so the largest was about 3.3 times the smallest ^[15]. The published paper reportedly extends the tests to seven models of up to 12 billion parameters ^[17]. That is about 29 times the preprint's ceiling ^[19]. Decrypt still calls it small by current standards ^[17], and its account does not give a hit rate for the larger set. The preprint's window was 300 tokens, a few short paragraphs ^[9]. A chat that tips after its 300th token was outside what those tests could measure ^[16].

The authors have in mind a phone or laptop running a model with no internet connection, companion chatbots included ^[7]. On that hardware their monitor runs in parallel with the model and flags when the predicted tipping point, n*, drops below a safety threshold ^[10]. The authors call the monitor low-cost ^[10]. An app builder would test that claim first, because the compute comes out of the same device that is running the chatbot.

This can play out a few ways. If the 12-billion-parameter tests land near 15 of 16, an offline team gets a warning it can compute locally. That matters because, as the authors argue, existing safety tools often depend on a cloud connection these models lack ^[6][10]. If the hit rate falls as models grow, the formula fits small models and loses value just as Decrypt expects on-device models to get more capable with better hardware ^[18]. A third possibility is that the formula holds but running the monitor in parallel costs more on a phone than the authors expect.

We think the work is strong enough for an on-device team to build a prototype monitor around. It is too thin for anyone to quote 94% as a deployment accuracy. The counter-case is that the authors tie tipping to how the attention head weighs accumulated context ^[4], which is a claim about how the model works and not about its size. The pair have also moved from one deliberately simplified attention head in their April 2025 paper ^[13] to six downloadable models with a single miss ^[8][14]. If the paper reports a hit rate for the 12-billion-parameter set well below 15 of 16, the case for prototyping is wrong. A rate at or above it would mean we were too cautious.