Open weights push protein-model safety off the API and onto the synthesis order

Model-level safety assumes a gate you control. For closed language models, the essay lists the usual gates: linear probes on activations, prompt refusals at the API, and unlearning to strip capabilities out of the weights ^[1]. Every one of those needs a request to pass through infrastructure you own. Open weights remove that request. Almost all of today's frontier protein models, including structure and function predictors and sequence generators, ship with open weights that anyone can download without touching an API ^[2].

So the essay pushes the control point somewhere it can still act. Two places, per the author. First, safeguards baked into the weights themselves, ideally ones that survive an attacker fine-tuning the model to remove them ^[4]. Second, the digital-physical interface, the step where a computational design becomes a physical sample ^[5].

That second step is more concrete, because biology has a bottleneck. To turn a designed sequence into a protein, you usually order the DNA from a third-party synthesis provider, which amplifies and expresses it in a host system ^[6]. That order is a real chokepoint. Stopping dangerous sequences there is, in the essay's words, the first line of defense ^[7].

The problem is what the providers screen with. Historically they compare an ordered sequence against known hazards and their closest database matches ^[8]. That matching catches dangerous sequences that resemble ones you have seen. A model that designs a functional sequence that has never existed defeats it. The essay's own argument is that sequence-based screening becomes unreliable exactly as de novo design gets good ^[9]. The proposed replacements are structure-based screens, which compare the predicted shape of an ordered protein against the shapes of known hazards, and interpretability features pulled out of the bio models themselves ^[10][11].

Now the incident. The essay reports that Anthropic revealed instances of malicious actors using Claude to design harmful proteins ^[12]. Be precise about it. The source here is a single essay on LessWrong, not Anthropic's own report, and the essay does not claim a working weapon. It says the probability of a catastrophic event, such as designing a biologically harmful virus, "seems unlikely to happen at least with current technology" ^[13].

The deployment constraint holds either way. If you release an open-weight protein model, the enforcement options are fixed before you ship: put the safeguard in the weights ^[4], or depend on whatever the synthesis provider screens for ^[5]. There is no API gate to add afterward ^[3].

The essay also notes that Dario Amodei, Sam Altman, and Elon Musk have called for AI development to slow down and be regulated ^[14]. The essay offers that as context, and it does not support the protein claim.