Product1 publisher2 min readPublished
Anthropic blocked a chikungunya grant-writing request as gain-of-function help
Anthropic says it stopped anonymous researchers who asked Claude to help write a gain-of-function grant application and to design novel venoms. Anyone shipping a science tool on a hosted model inherits that judgement call.
The Product Desk · Product desk
What happened
- Anthropic has said anonymous scientists tried to use its flagship Claude model for research that could have turned deadly, and the company blocked the attempts.
- One blocked request asked Claude to help with a grant application for gain-of-function research on the mosquito-borne Chikungunya virus, and Anthropic did not identify the scientists involved.
- The company says there is no evidence the scientists were actually trying to cause harm.
- Fast Company writes that dressing a harmful request up as a request for other help, such as writing a grant application, is a classic jailbreak against a model's guardrails.
- Fast Company also writes that Anthropic admits it could previously hide behind the fact that its models simply were not that good at science.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- constraint Legitimate toxin and virology work on a hosted model now sits behind a per-request judgement made by the vendor, so a virologist's ordinary document is the thing that can stop the workflow.
- contradiction Anthropic offers the blocked cases as evidence its controls work, while Fast Company writes there is no way to know how many similar requests passed the filters, so nobody outside the company can size the miss rate.
- cost Over-blocking is expensive on both sides: researchers lose working days, and Fast Company notes Anthropic risks sending big customers to OpenAI.
A grant application reads like ordinary science. The aims section states in plain language that the team intends to make a pathogen more deadly or easier to spread, because that is what gain-of-function research is [4]. Anthropic's report said the choice of pathogen is what made this case dangerous: "Because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak" [5].
The second incident is the one to read twice if you ship a science tool. Anthropic says users asked Claude to help develop "novel venoms and toxins" [6]. The company's own example of why that work is legitimate is Botox, which comes from a deadly bacterial toxin and is used "to treat migraines, spasticity, and wrinkles" [7]. Both of the incidents it described arrived wrapped in ordinary research tasks [8]. Fast Company describes the squeeze this puts Anthropic in: the company does not want to shut down promising research or drive big customers to a competitor like OpenAI [9].
A request that looks like legitimate science can be refused, and the vendor may write it up afterwards. Fast Company's account of the report covers the blocking and not the enforcement; it does not mention suspended accounts, how the requests were flagged, or any change to the product [10].
Teams tell themselves their research users sanitize the prompt and keep the sensitive step out of the chat window. Those users paste the live document the night before the deadline, transmissibility paragraph included, because that paragraph is the reason the grant exists.
What matters is whether the task contains a dual-use step, meaning a pathogen, a toxin, or a synthesis route, and whether a refusal at that step merely annoys the user or ends the session. If the task has no dual-use step, a hosted model needs no special handling. Where the step exists and the user can wait, the thing to settle before rollout is a human route: a named contact at the vendor, plus an appeal that comes back while the user is still working. Where a refusal ends the session, the alternative Fast Company points to is open weights, which it says run only a few months behind today's top frontier models and often include almost no guardrails [11]. Self-hosting moves the screening judgement, and the cost of getting it wrong, onto your own team.
What to watch
- Whether Anthropic publishes counts or rates for blocked biology requests instead of selected examples.
- Whether Anthropic documents an appeal path or verified-researcher route for dual-use prompts that get refused.
- Whether the refused workloads reappear on the open-weight models Fast Company describes as shipping with almost no guardrails.