SecurityNot yet confirmed elsewhere1 publisher2 min readPublished Updated
George Washington University formula predicts when a chatbot's attention tips into harmful output
George Washington University researchers say a formula predicts how many good outputs a chatbot gives before its first bad one, across seven open-weight models. Their proposed check sits inside the model, so offline AI companions on personal devices, with no cloud filter and no way to patch weights, would need it before they ship.
The Watch · Security desk
What happened
- Johnson and Huo argue that conversation context competes inside the attention head with rival output basins, and accumulated context can push the model past a tipping point.
- The formula was tested on models from three independent groups, sized from 124 million to 12 billion parameters, and results matched its predicted immediate and delayed tipping regimes.
- Johnson said his lab has already added a warning check to open-source models but cannot get inside the closed models built by OpenAI and Anthropic.
- The researchers focused on offline AI companions because half the world's population carry devices able to run them with limited security, SecurityWeek reported.
Why it matters
- constraint Closed chatbots get the warning check only if their vendors build it in, because teams building on those models have no access to the attention layer to add it themselves.
- exposure If Johnson's contagion account holds, every agent in a connected group carries the risk from whichever one tips first, and no human attacker is needed to start the spread.
- constraint Testing stopped at 12 billion parameters on open-weight models, so applying the formula to larger closed chatbots is an extrapolation until someone tests it there.
Inside a transformer, the attention head decides which earlier tokens matter most while the model processes the current one, and that decision sets up the next token [7]. Johnson and Huo, working at George Washington University, put the tipping point in that step [1][10]. Once a first bad output appears, it starts shaping later tokens in unintended ways, and the model can be classed as misaligned [11].
The input that moves a model toward the tip is a prompt. One malicious prompt can tip it at once, and a run of poor prompts can tip it later [9]. Under the paper's argument, thoughtless prompts hasten the slide as well as malicious ones [9].
Because the formula yields a number, a check can test against it before the model emits its next token [10]. Johnson proposes exactly that. "A simple warning light placed within the AI before it produces its next output. This is easy for AI companies to insert," he told SecurityWeek [5].
The researchers chose on-device companions as their test bed. Those deployments run without cloud safety filters, real-time telemetry or live monitoring, and their weights cannot be patched once deployed [3]. SecurityWeek called that a good setting for studying a chat model's behavior when it is left to its own devices [3]. The test models varied widely in size. The largest was roughly 97 times the smallest [13].
Johnson extends the effect to groups of AI agents. "An AI-agent ('Patient Zero') tips to generating undesirable output, with no human required," he told SecurityWeek [15]. Other agents then receive that output, he said, and "This tipping propagates among the AI agents who have contact with each other" [15]. SecurityWeek's account of the testing covers the seven single models and does not describe a multi-agent experiment behind that claim, or report how often the warning light fires on a conversation that was never going to tip [4].
Bri Frost, director of product management at Cloud Range, made a related point about agents in separate comments. "An agent doesn't need bad intent to create risk. It just needs a goal, access and no clear sense of where its boundaries are," he said [12].
What to watch
- A published false-alarm rate and lead time for the in-model warning light, the figures that decide whether it can gate output on a device.
- Any vendor of a closed or on-device model saying it has added an attention-based tipping check before release.
- A multi-agent experiment testing Johnson's account that one tipped agent spreads bad output to the agents it talks to.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence35
- Adoption5
- Hype gap+35
- Incentives
- Insufficient
- Confidence35
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Researchers from George Washington University published a paper examining whether the time and cause of AI going rogue can be predicted and, if predicted, prevented.
- [2]
Since 50% of the world's population carry devices that can run personal AI companions with no internet connection and limited security, the researchers focused their research on these companions.
- [3]
On-device AI companions lack cloud-based safety filters, real-time telemetry, live monitoring, or the ability to patch weights once deployed; SecurityWeek described this as a good test bed for analyzing chat-style transformer behavior when left to its own devices.
- [4]
The tipping-point formula was tested across seven open-weight transformer models built by three independent groups, ranging from 124 million to 12 billion parameters; results showed consistent alignment with predicted immediate versus delayed tipping regimes.
- [5]
"A simple warning light placed within the AI before it produces its next output. This is easy for AI companies to insert," Neil Johnson told SecurityWeek.
- [6]
"We have already inserted this in the open source models in our lab, but obviously we cannot get inside OpenAI and Anthropic's closed AI models to do this," Johnson told SecurityWeek.
- [7]
The attention head is the computational component that determines which earlier tokens are most relevant when processing the current token; that decision lays the foundation for the next token.
- [8]
The paper's primary argument is that output can slip from good to bad due to competition in the attention head between the conversation's context and competing output basins; accumulated context can gradually shift attention toward an undesirable basin until a tipping point is crossed.
- [9]
Both thoughtless and malicious prompts can hasten the AI's slippage into bad outputs, either immediately (one bad prompt) or delayed (the accumulated effect of poor prompts).
- [10]
Researchers Neil Johnson and Frank (Yingjie) Huo developed a mathematical formula to estimate the tipping point, represented by the number of good outputs that occur before the first undesirable output appears.
- [11]
Once the first bad output occurs, it starts to influence future tokens in unintended ways, alignment with purpose is lost, and the AI can be categorized as misaligned.
- [12]
"An agent doesn't need bad intent to create risk. It just needs a goal, access and no clear sense of where its boundaries are," said Bri Frost, director of product management at Cloud Range.
- [13]
The largest test model (12 billion parameters) was roughly 97 times the size of the smallest (124 million parameters).
- [14]
Because the proposed warning check sits inside the model and on-device weights cannot be patched after deployment, an offline companion would need the check in place before it ships.
- [15]
"An AI-agent ('Patient Zero') tips to generating undesirable output, with no human required. Other AI-agents then receive that undesirable output. This tipping propagates among the AI agents who have contact with each other" - Neil Johnson to SecurityWeek.
Sources
1 independent publisher whose own reporting we read for this story.
- securityweek.comFormula Predicts When AI Chatbots Are at Risk of Turning Bad
1 article · October 8, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Transformer attentionFollow
- On-Device AIFollow
- AI alignmentFollow
- AI Agent SafetyFollow