Open-weight refusal is one line away from off. Here is how we made it cost more.
The uncomfortable fact
When you release an open-weight instruction-tuned model, you are also releasing the off switch for its safety behavior. Arditi et al. (2024) showed that refusal is mediated by a single linear direction in activation space. You recover it with a difference-in-means between harmful and harmless prompts, and once you have it, a rank-1 ablation strips refusal while leaving the rest of the model essentially unchanged.
That is what bothered us. The safety property that costs a full post-training run to install comes off for the price of one direction (K=1). We wanted to know whether you could make that attack meaningfully more expensive without retraining from scratch and without making the model worse at its job.
Interactive explainer (47 seconds)
The wrong instinct, and the right one
The tempting move is to hide the direction or to make it bigger. Both are dead ends: an attacker recomputing difference-in-means does not care how large the direction is, and obfuscating one direction just produces a slightly harder-to-find single direction. The geometry is the problem, not the magnitude.
So the goal flips. Do not protect one direction. Spread the refusal signal across many directions so that no low-rank ablation can remove it. If the safety behavior is genuinely distributed over a 16-dimensional subspace, an attacker now has to ablate at least 16 directions, and each one they remove risks damaging general capability.
The method
We post-trained Llama-3.2-1B-Instruct with two objectives working together, in PyTorch:
- Class-conditional mean and covariance matching. We shape the activation statistics of the harmful and harmless classes so the discriminative signal is no longer concentrated in a single difference-in-means direction. This is the part that actually moves the geometry.
- Temperature-scaled KL distillation from a frozen copy of the original instruct model. This is the leash: it keeps general behavior and helpfulness close to the original while the matching objective rearranges the safety subspace.
The framing we kept coming back to: the matching loss pushes the safety signal outward across directions, and the distillation loss holds everything else in place. Without the leash you can spread the signal and wreck the model; without the push you keep a clean model with a single fragile direction.
Did it work
The headline number is the rank K of the linear ablation needed to break refusal. On the baseline instruct model it is K=1. On the defended model it is K≥16, at least a 16x increase in the attacker's budget, with baseline refusal and general behavior preserved.
We did not want to trust a single behavioral metric, because metrics going up is exactly what a model that learned to obfuscate would also produce. So we evaluated two ways: behavioral checks that the model still refuses harmful requests and still complies with benign ones, and linear-probe diagnostics that measure how many directions actually carry the safety signal. The probes are the important part, they confirm the signal is genuinely redistributed rather than hidden in one place.
What we took away
- Single-direction safety is a fragility, and the fix is to change the representation geometry, not to patch the behavior.
- You can buy a real increase in attack cost without a fresh pre-training run and without a utility tax you can measure.
- Ship the defense and the evaluation harness as one thing. The probe diagnostics are what let you claim the signal moved, not just that a number went up.