Skip to content

Security1 publisher2 min readPublished

An 11-model probe puts the "this is impossible" signal 85 degrees from the refusal direction

Yucheng Du and Xiyang Hu report a single direction that separates answerable from unanswerable math and code prompts at 0.939 mean AUC across 11 models, and its mean cosine with the canonical safety-refusal direction is 0.087.

The Watch · Security desk

Illustration accompanying An 11-model probe puts the "this is impossible" signal 85 degrees from the refusal direction

What happened

  • Yucheng Du of USC and Xiyang Hu of Arizona State released code and data for an EMNLP 2026 main conference paper on why models answer structurally unanswerable math and code questions.
  • A single null-space mean-difference direction separated answerable from unanswerable prompts at mean AUC 0.939 across 22 model-domain cells, on a grid of 11 models from 1.7B to 70B parameters.
  • Steering activations along the direction at generation time raised gated delta-G by 33 to 52 percentage points on the four-anchor math and code intervention grid.

Compiled by The WatchSomething wrong?How this is made

Why it matters

  • constraint Refusal training works on an axis about 85 degrees from the one carrying impossibility, so tightening a model's refusals on unsafe prompts does nothing for impossible ones.
  • capability Activation steering on the impossibility direction is a control that bypasses safety tuning entirely, and it moved behavior by tens of points in the authors' own grid.
  • cost Confirming the published numbers takes minutes against the shipped aggregates; contesting them at the tensor level means pulling the omitted 151 GB signal tree and paying for GPU time.
  • precedent Anyone building an eval suite now has a probe for a failure mode that refusal-rate metrics miss.

The direction is one-dimensional, fitted in the null space as a mean difference between hidden states on matched answerable and unanswerable prompts [5]. Cosines convert to angles. The mean cosine against the Arditi et al. refusal direction works out to about 85 degrees, and the reported range of 0.020 to 0.130 spans 82.5 to 88.9 degrees [6][1]. The behavior-defined invalidity-aware direction, fitted in-domain from what the model does rather than from geometry alone, sits at roughly 66 degrees from the same impossibility direction [7][2].

The base/instruct comparison is the part that bears on guardrail tuning. The largest change any of the six pairs on math800 showed, +0.110, is the same size as the entire spread of cosines measured across the main grid [8][4]. The release says instruction tuning "modulates the angle in a low-cosine regime" it does not produce [9].

Steering is where the paper claims causality. Activation steering along the impossibility direction at generation time changed invalidity-aware behavior on a four-anchor math and code grid, and a 16-model sweep reproduced the same footprint [10].

The labels behind the shipped intervention aggregates are LLM-assisted under a fixed rubric: nine cells in the four-anchor grid use provisional audit-subset fills, three Qwen3-8B cells use candidate-label passthrough, and a supplementary Gemma-3-12B code cell also uses fills [11]. Twelve labeled cells carry the intervention numbers [5], and the repository says the provenance labels do not claim completed human adjudication [12]. The release limits its own scope as well: both domains are structural impossibility in math and code [4]. The ACL Anthology record is pending [2].

Checking the headline figures is cheap. A verification script reads the shipped aggregate artifacts and provenance metadata, and the lightweight smoke test finishes in under five minutes [14]. Recomputing representations from scratch requires the omitted 151 GB signal tree, and the intervention pipeline needs GPU access [13]. The 5-to-10-dimensional subspace-overlap robustness check ships as frozen JSON that can be read without the tensors, though rerunning the scripts needs the raw signals and, for principal angles, the model weights [15].

For anyone reading a refusal-rate number off a model card, this evidence says the fabrication-on-impossible-input failure is carried on a different axis, roughly perpendicular to the one refusal training addresses [6][1]. The paper ships both a probe for that axis and a steering lever on it [5][10]. Both have been tested only in math and code [4].

What to watch

  • Whether human adjudication replaces the provisional audit-subset fills and Qwen3-8B candidate-label passthrough in the twelve labeled intervention cells.
  • Whether the near-orthogonality survives outside structural impossibility in math and code.
  • Whether the camera-ready and the pending ACL Anthology record keep the 0.939 AUC and 0.087 cosine figures.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories