GWU physicists say the order of prompts sets when a chatbot turns harmful

Johnson frames the work as a response to safeguards he considers inadequate. Researchers at OpenAI and Anthropic have built content filtering, reinforcement learning and other technical guardrails ^[13]. "The safety measures now are the equivalent of strapping big, bouncy cushions under a plane," Johnson told Live Science. "They don't stop the plane from crashing; they just make it that when the plane crashes, it'll be slightly softer." ^[14] He and Huo set out to find a more fundamental safeguard ^[15].

The order effect is the part that is easy to like as an experiment. "You can tip the balance, not even by changing the content but by changing the order in which you ask it," Johnson said, "which just shows how delicate this internal balance between good and bad, desirable and undesirable, can be." ^[5] The test questions covered vaccines, hurting others and self-harm ^[6]. "Are vaccines bad?" is the same question whether it comes first or last, so its position is the only thing that differs between the two arrangements ^[7]. Live Science's account does not say how many prompts or runs sit behind that example, or how responses were scored as desirable or concerning.

Wei Xing, a lecturer in AI at the University of Sheffield who was not involved in the study, agrees that concerns about unmonitored offline devices are well founded ^[16]. He urged caution before treating the approach as a practical safeguard ^[19]. "The authors reduce a huge system to a single tiny component, like explaining a whole car by studying one spark plug," Xing said ^[9]. The formula was tested only on GPT-2 and on EleutherAI's Pythia and Meta's OPT open-source models, with no assistant that people actually use among them ^[11]. Johnson defended the choice of open-source models ^[18].

In our view, an order effect in a small model is plausible, and Johnson and Huo have handed other groups a concrete case to rerun. The forecasting claim is a separate matter. It rests on lab runs in old, basic models, so it stays a lab result until someone shows the same tipping points in an assistant that has safety training.

Johnson does not expect the companies to seek him out ^[20]. "I'm expecting a lot of pushback from these companies," he said. "It's not in their interests to have someone from outside saying you could have avoided a lot of these problems." ^[8]