Security Not yet confirmed elsewhere 1 publisher 2 min read Published Aug 28, 2026, 3:48 PM PDT

Unit 42 locates Qwen3-4B's refusal template in 50 of 350,208 neurons

Perturbation probing needs the weights, which puts the finding on teams running open-source checkpoints rather than on API tenants. The same 50-neuron toolkit that breaks a refusal template also repairs other behavior.

The Watch · Security desk

How we use AI Send a correction

What happened

Unit 42 published a method it calls perturbation probing, which uses two forward passes per prompt to pin a target behavior such as harmful-request refusal to specific feed-forward neurons.
On the open-source Qwen3-4B model, the method located the safety refusal template in 50 neurons out of 350,208, about 0.014% of the feed-forward layer.
On Qwen3.5-2B, editing 20 neurons dropped false agreement with users in multi-turn conversation from 36.7% to zero across 30 questions.

Why it matters

Who's exposed exposure Risk here sorts by who holds the checkpoint: anyone with read access to an open-weight model inherits the ability to strip its refusal layer, while API-only tenants have no path to the neurons at all.
What it limits constraint Refusal behavior concentrated this tightly is hard to cite as a control in an audit, because an ordinary fine-tune can move it and nothing in the usual deployment pipeline measures that drift.
Who has to decide decision Buyers can now ask a model vendor for a fragility number computed before any red-team campaign, which changes what counts as an adequate answer about alignment robustness.
What's now possible capability Neuron-level editing becomes a behavior knob rather than only a removal tool, putting small unaudited behavioral patches within reach of any team that holds weights.

Next story · Security · 19h ago

FortiManager flaw CVE-2024-47575 exposed configs and private keys for 500+ FortiGate firewalls

Confirmed2 publishers

Perturbation probing's threat model is narrower than the headline number suggests. It operates on weights: two forward passes per prompt, reading and then disabling specific feed-forward neurons ^[1]. An API tenant calling a hosted model cannot reach a neuron. A team that pulled Qwen3-4B down, fine-tuned it, and put it behind a support chatbot owns all 350,208 of them ^[6], and so does anyone who gets read access to that checkpoint.

Cost is trivial. Two passes per prompt across the 520-prompt benchmark Unit 42 used comes to 1,040 forward passes ^[15], on a 4B model. Nothing there requires a research budget.

The part that lands on defenders who will never be attacked this way is Unit 42's aside that a normal optimization run could shift the same layer ^[5]. If the refusal template sits in roughly one neuron in 7,004 ^[14], routine fine-tuning can degrade it as a side effect, with no adversary involved and no signal in any deployment pipeline that currently checks for it.

The measurement here is specific: ablating the 50 neurons changed the response format on 80% of the 520 harmful prompts ^[2], which is 416 prompts ^[11], and the result held on 200 prompts of a second benchmark ^[2]. Format change means the refusal template broke. The post does not report a harmful-completion rate for the ablated model. Bypass was the subject of the earlier logit-gap steering work, which got past guardrails by closing a measurable gap in the model's output scores ^[8]. This paper answers where the behavior lives, and stops there.

The sycophancy result is directionally clean and thin. 36.7% of 30 questions is 11 questions ^[12]; 20 neurons took those 11 to zero on Qwen3.5-2B ^[3]. Thirty questions is a demonstration, not a rate.

The same lever pushes the other way. Amplifying 10 identified neurons moved factual self-correction from 52% to 88% on 200 TruthfulQA prompts with no retraining ^[4], which is 104 prompts to 176 ^[13].

The durable contribution is the FFN/Skip ratio: seconds to compute per model, and across the 13 models tested it explained 81% of the variance in how much a small targeted change moved safety behavior ^[16]. That is a procurement-usable number if the per-model values get published, and the blog post names only Qwen3-4B and Qwen3.5-2B, with the method in an arXiv paper ^[9]. The recommendation stacked on top, external content filters and runtime guardrails rather than trust in trained-in refusal ^[7], is standard defense in depth, and Unit 42 closes with its own product, Prisma AIRS Runtime Security, as the tool it points to for that job ^[10].

What to watch

Whether the arXiv paper publishes per-model FFN/Skip ratios for all 13 models tested, which is what would make the score usable in model selection.
Whether an independent group reproduces the 50-neuron ablation and reports harmful-completion rates rather than response-format change.
Whether fine-tuning platforms start measuring refusal drift after a customer training run, given Unit 42's claim that ordinary optimization can move the same layer.

Clarity's read

What the record supports and how the coverage leans. The claims behind it follow.

Reality

Evidence42
Adoption: Insufficient
Hype gap+30
Incentives80
Confidence45

Why these scores

Claim ledger

Ranked by verification strength, evidence, and original report placement.

[1]
Unit 42 introduced a method called perturbation probing that uses two forward passes per prompt, at significantly lower computational cost than prior approaches, to identify the small set of feed-forward neurons causally responsible for a targeted behavior such as refusing harmful requests.
Reported Supported Source: Unit 42 (Palo Alto Networks)2 sources— create a free account to open them View cited source
[2]
Removing those 50 neurons changed the response format on 80% of 520 standard harmful-prompt benchmark items, and the result was replicated on 200 prompts of a second standard benchmark.
Reported Supported Source: Unit 422 sources— create a free account to open them View cited source
[3]
On the smaller model Qwen3.5-2B, 20 neurons were enough to stop the model from falsely agreeing with users in multi-turn conversations, dropping that behavior from 36.7% to 0% across 30 questions.
Reported Supported Source: Unit 422 sources— create a free account to open them View cited source
[4]
Amplifying 10 identified neurons on a small model improved factual self-correction from 52% to 88% on 200 TruthfulQA prompts without any retraining.
Reported Supported Source: Unit 422 sources— create a free account to open them View cited source
[5]
Unit 42 states that refusal behavior lives in a thin template layer that an attacker able to manipulate internals could disable, and that even a normal optimization run could shift it.
Reported Supported Source: Unit 422 sources— create a free account to open them View cited source
[6]
On the open-source LLM Qwen3-4B, 50 neurons out of 350,208 feed-forward neurons, about 0.014%, control the safety refusal template.
Reported Supported Source: Unit 42View cited source
[7]
Unit 42 calls reliance on that thin layer alone the LLM analog of relying on a single perimeter firewall, and argues for defense in depth with external content filters and runtime guardrails layered on top of the base model's training.
Reported Supported Source: Unit 42View cited source
[8]
Unit 42's earlier logit-gap steering research showed that the safety guardrails of an aligned LLM can be bypassed by closing a measurable gap in the model's output scores.
Reported Supported Source: Unit 42View cited source
[9]
The full method is published on arXiv as "Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs"; the blog post names only Qwen3-4B and Qwen3.5-2B among the models tested.
Reported SupportedView cited source
[10]
The post closes by stating that for organizations deploying LLMs today, Palo Alto's Prisma AIRS Runtime Security provides the external content filters and inline guardrails the thin template layer cannot.
Reported SupportedView cited source
[11]
80% of the 520-prompt harmful benchmark is 416 prompts on which the response format changed.
Derived Supported2 sources— create a free account to open them
[12]
36.7% of 30 questions is about 11 questions, so the 20-neuron edit eliminated false agreement on roughly 11 items.
Derived Supported2 sources— create a free account to open them
[13]
On 200 TruthfulQA prompts, 52% to 88% is a move from 104 prompts to 176, an increase of 72.
Derived Supported2 sources— create a free account to open them
[14]
50 controlling neurons out of 350,208 feed-forward neurons is roughly one neuron in 7,004.
Derived Supported1 source— create a free account to open them
[15]
Running the two-pass method over the 520-prompt benchmark costs 1,040 forward passes.
Derived Supported1 source— create a free account to open them
[16]
The same computation yields an FFN/Skip ratio, a single number computable in seconds per model, which across the 13 models tested explained 81% of the variance in how vulnerable each model's safety behavior was to a small targeted change.
Reported Insufficient Source: Unit 422 sources— create a free account to open them View cited source

Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.

Add to ChatGPT Add to Claude

Sources

1 independent publisher whose own reporting we read for this story.

unit42.paloaltonetworks.com
1 article · August 28, 2026
Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

Ask Clarity

Have a question this piece left open? The answer comes from its reporting and claim ledger — nothing else.

Try

Clarity analysis

How the coverage leans and what the record actually supports. Read the methodology

Perspective Coverage

How much of the coverage speaks to builders, operators and investors. It measures emphasis, not whether the story is right.

Builder

The engineering content here is a cheap causal-attribution recipe: two forward passes per prompt, ablate, measure. What makes it interesting for anyone fine-tuning open checkpoints is not the attack framing but Unit 42's own aside that a normal optimization run could shift the same template layer — meaning a routine post-training pass can move refusal behavior without anyone editing a neuron on purpose. The mirror result is just as useful: ten amplified neurons moved factual self-correction from 52% to 88% on TruthfulQA with no retraining. Both directions depend on having the weights, and the paper on arXiv is where the actual reproducible detail lives; the post is a summary.

Reality

Evidence, adoption, hype gap, incentives and confidence, each read independently from the current record on the scale shown beneath it. How these are measured.

Evidence42: 0100
Precise numbers, one hand holding the ruler
The figures are unusually specific for a vendor blog — denominators, sample sizes, before-and-after rates — and the method is on arXiv for anyone who wants to check it. But every measurement in this story was taken by Unit 42, reported in aggregate, with generations withheld and eleven of the thirteen tested models unnamed. Specificity is not the same as verification, and nothing here has been through a second pair of hands.
AdoptionInsufficient: No user but the author
Unit 42 recommends that teams add fragility diagnostics to their evaluation pipelines, and that is where the trail ends. There is no disclosed use of perturbation probing or the FFN/Skip ratio by anyone outside the team that invented them — no model vendor citing the score, no security team reporting a gate built on it. Self-run experiments demonstrate a method; they do not measure uptake, so we leave this unscored rather than dress the benchmarks up as traction.
Hype gap+30: −100 understated0 aligned+100 overstated
Headline outruns the measurement
"Fifty of 350,208 neurons control safety" is the sentence that travels, and the underlying measurement is softer: 80% of prompts showed a changed response format, not 80% successful jailbreaks, and the crispest result — 36.7% to zero — rests on thirty questions. The scoping omission adds to the gap. Perturbation probing needs the weights, so this is a finding about self-hosted open checkpoints, yet the framing addresses LLM safety in general. The overstatement is in the reach, not in the arithmetic.
Incentives80: 0100
The research and the remedy share a vendor
Palo Alto Networks funded the study that concludes base-model alignment is structurally insufficient, and the final paragraph sells the layer that fixes it — Prisma AIRS Runtime Security, plus a Unit 42 AI Security Assessment. That does not make the neuron counts wrong; it does mean the argument's shape was commercially useful before it was tested. The countervailing signal is a real one: the method is on arXiv, harmful generations are withheld, and the same toolkit is shown repairing behavior as well as breaking it, which is not what pure fear marketing looks like.
Confidence45: 0100
Internally consistent, externally untested
We can say with confidence what Unit 42 claims and how it frames the stakes; the numbers are coherent, the arithmetic holds, and the prior logit-gap work gives the argument a track record. What we cannot say is whether fifty neurons really carry Qwen3-4B's refusal, because a single interested publisher is the whole record and the mechanism sits behind a paper this reporting does not open.

Topics and entities

Follow any of these and your For You feed starts watching them — no settings page required.

Topics

AI Runtime GuardrailsFollow
Open-Weight Model RiskFollow
LLM Alignment RobustnessFollow
Mechanistic interpretability Follow
Vendor-published security research Follow

Entities

Unit 42 Follow
Palo Alto Networks Follow
Perturbation ProbingFollow
FFN/Skip RatioFollow
Qwen3-4B Follow
Qwen3.5-2BFollow
TruthfulQAFollow
Logit-Gap SteeringFollow
Prisma AIRS Runtime SecurityFollow