🤖 machine learning

Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks

This paper reveals that the low-dimensional, brittle nature of refusal mechanisms in aligned language models stems from repetitive refusal prefixes during training, and demonstrates that employing diverse refusal prefixes can increase the stable rank of activation updates, thereby making refusals more robust against vector ablation attacks.

Original authors: Andrey Labunets

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrey Labunets

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems are becoming increasingly capable, able to write code, diagnose medical conditions, and generate creative content. With this power comes a significant responsibility: ensuring these machines do not learn to generate harmful instructions, such as how to build weapons or create dangerous biological agents. To prevent this, developers train models to recognize unsafe requests and politely decline them, a process known as refusal training. However, a troubling vulnerability has emerged. Researchers have discovered that in many advanced models, this refusal behavior is not a complex, distributed network of defenses, but rather a single, narrow path in the model's internal thinking space. If an attacker can identify and block this specific path, the model can be tricked into ignoring its safety rules and complying with dangerous requests. This weakness suggests that the very way safety is taught might be creating a single point of failure.

A researcher at UC San Diego set out to understand why this fragile, single-path structure forms and whether it can be made more robust. They focused on a specific type of AI model to trace the origins of its refusal behavior back to the training process itself. Their investigation revealed that the geometry of the refusal mechanism is a direct reflection of how the model was taught to say no. Specifically, they found that when a model is trained to refuse a harmful request, the changes it makes to its internal state are heavily influenced by the very first word it uses to begin its refusal. If the training data forces the model to start every refusal with the same phrase, such as "I am sorry," the model's internal adjustments become highly concentrated. This concentration creates a low-dimensional structure, meaning the refusal behavior relies on a very small number of directions in its internal space. Because the behavior is so concentrated, it is easy for an attacker to find that single direction and neutralize it, effectively jailbreaking the model.

The researcher tested this theory by examining how different training patterns affected the model's resilience. They observed that when the refusal training data was repetitive, with the model learning to start every refusal with the same token, the resulting safety mechanism was indeed brittle. The internal changes were so focused that removing a single vector could disable the refusal capability entirely. However, when they introduced diversity into the training process, requiring the model to learn from refusals that began with many different words and phrases, the outcome changed dramatically. By forcing the model to distribute its learning across a wider variety of starting points, the internal changes became more spread out. This increased the effective rank of the model's internal adjustments, making the refusal mechanism less dependent on any single direction.

To verify that this diversity actually strengthened the model, the researcher conducted controlled experiments where they fine-tuned the model on datasets with varying levels of refusal diversity. They found a clear pattern: models trained with diverse refusal starts were significantly harder to break. When subjected to the same type of attack that easily bypassed the repetitive models, the diverse models maintained their refusal behavior much more effectively. The attack, which worked by projecting out a single direction, failed to remove the refusal capability because the safety signal was now spread across many directions rather than concentrated in one. The researcher measured this using a concept called stable rank, which essentially counts how many independent directions are needed to describe the model's behavior. They found that higher diversity led to higher stable ranks, and higher stable ranks correlated directly with a smaller drop in safety performance when attacked.

The study also explored how these changes propagate through the layers of the model. They discovered that the diversity introduced at the very beginning of the training process, in the form of varied first tokens, influenced the model's internal state all the way through its deepest layers. The changes in the early layers, where the model first processes the request, were amplified as they moved through the network, resulting in a more robust refusal mechanism at the final decision point. This suggests that the way safety data is structured at the token level has a profound and lasting impact on the model's overall security architecture. The researcher noted that while this approach does not guarantee immunity against all possible attacks, it provides a simple and effective lever to harden models against a specific and dangerous class of vulnerabilities.

The findings offer a new perspective on AI safety, shifting the focus from just what the model is taught to how it is taught. The research suggests that the brittleness of current safety alignments is not an inevitable flaw of the technology, but a consequence of repetitive training patterns. By diversifying the way models learn to refuse, developers can create safety mechanisms that are more distributed and therefore more difficult to dismantle. This work provides a concrete, measurable method for improving the robustness of AI systems, showing that the path to safer models may lie in the simple act of varying the vocabulary of refusal. The study concludes that understanding the geometric relationship between training data and model behavior is essential for building defenses that can withstand the evolving tactics of those who seek to misuse powerful artificial intelligence.

Technical Summary: Refusal Geometry Reflects Refusal Training

Problem Statement

As AI models acquire dual-use capabilities (e.g., exploit generation, bio-weaponization assistance), safety alignment becomes critical. Standard alignment techniques train models to refuse harmful instructions. However, recent work has identified a vulnerability: in white-box settings, refusal behavior in aligned language models can be mediated by a single activation direction or a low-dimensional refusal subspace. Ablating this direction (removing its projection from activations) effectively "jailbreaks" the model, suppressing refusals while preserving other capabilities.

Despite the prevalence of this phenomenon, the mechanistic origin of why safety-critical refusal features concentrate into such low-dimensional structures across diverse harmful prompts remains unclear. Furthermore, it is unknown whether this brittleness is an inherent property of the model architecture or a consequence of specific training dynamics, and whether it can be mitigated.

Methodology

The author conducts a case study on the OLMo-2-0425-1B-Instruct model, combining analytical derivations with empirical validation across frozen-model analyses and controlled synthetic fine-tuning.

Analytical Framework

The paper posits that the geometry of refusal residuals (the activation differences between harmful and benign prompts) is directly shaped by the gradients of the refusal training process.

Gradient-to-Activation Mapping: The author analyzes cross-entropy losses on the first tokens of refusal completions. They derive that the gradient-induced activation updates ( $G$ ) resulting from these losses should geometrically correspond to the refusal residuals ( $\Delta H$ ). Specifically, they hypothesize that the refusal subspace is approximated by the sign-reversed affine subspace of these gradient updates.
Stable Rank and Concentration: The core theoretical contribution is linking the stable rank (a measure of effective dimensionality) of the refusal mechanism to the diversity of refusal training data.
- Lemma 3.1: The stable rank of the one-hot matrix of refusal first tokens ( $E_r$ ) is inversely proportional to the maximum frequency of any single token. If all refusals start with the same token (e.g., "I'm sorry"), the stable rank is 1. If tokens are diverse, the stable rank increases.
- Propagation: The author defines stable rank transfer factors to bound how the stable rank of gradients propagates through transformer layers. They show that under idealized conditions, the stable rank of the final layer's gradients (driven by token diversity) propagates back to earlier layers, potentially amplifying the effective rank of the refusal mechanism.

Experimental Setup

Models: OLMo-2-0425-1B and its checkpoints (Base, SFT, DPO, RLVR1, Instruct).
Datasets: AdvBench, WildJailbreak, XSTest, and a controlled CAMEL chemistry dataset for fine-tuning.
Attack: Difference-in-means single-vector ablation. The attacker estimates a refusal vector $\rho = \mu_{harm} - \mu_{benign}$ and removes its projection from activations at runtime.
Metrics: Refusal rate, harmfulness rate, refusal delta (change in rate post-ablation), stable rank of residuals ( $\Delta H$ ) and gradients ( $G$ ), and subspace overlap metrics.

Key Contributions and Results

1. Training-Origin Geometry

The study confirms that refusal training gradients directly shape the refusal subspace.

Alignment: The mean of gradient-induced activation updates ( $G$ ) is negatively aligned with the difference-in-means refusal vector ( $\rho$ ), consistent with gradient descent moving activations opposite to the loss gradient.
Subspace Overlap: The principal directions (right singular vectors) of the centered gradient updates ( $G_c$ ) and the centered refusal residuals ( $\Delta H_c$ ) exhibit significant overlap, particularly in middle and late layers.
Rank Correspondence: The stable ranks of $G$ and $\Delta H$ are closely matched, supporting the hypothesis that the refusal mechanism is a direct consequence of the training updates.

2. Prefix Concentration and Spectral Collapse

The paper identifies repetitive refusal starts (e.g., all refusals beginning with "I") as a primary mechanism for low-rank refusal representations.

Token Diversity: When refusal completions share the same first token, the target matrix $E_r$ has a stable rank of 1. This concentration propagates through the network, resulting in low-stable-rank gradients and activation updates.
Vulnerability: Low stable rank implies that the refusal mechanism is concentrated in a single effective direction, making it highly susceptible to single-vector ablation attacks.

3. Refusal Diversity as a Hardening Lever

Through controlled experiments, the author demonstrates that increasing the diversity of refusal first tokens raises the stable rank of the refusal mechanism and weakens ablation attacks.

Frozen Model Analysis: By varying the first tokens of refusal completions on a fixed set of harmful prompts, the author showed that increasing the number of distinct first tokens (from 1 to 16) increased the stable rank of the refusal residuals ( $\Delta H$ ) and gradient updates ( $G$ ).
Ablation Resistance: Subsets with higher stable ranks exhibited smaller "refusal deltas" (the drop in refusal rate after ablation). A higher-rank residual subspace is harder to suppress with a single vector.
Controlled Fine-Tuning: The author fine-tuned the model on benign chemistry prompts paired with refusal completions of varying token diversity. Models trained with diverse refusal starts achieved comparable baseline refusal rates but showed significantly reduced vulnerability to single-vector ablation compared to models trained with repetitive starts.

4. Cross-Layer Stable Rank Transfer

The study introduces stable rank transfer factors to quantify how rank changes across layers.

Amplification: In many settings, the stable rank of gradients amplifies as they propagate backward from the last layer to middle layers (e.g., a factor of ~2 from layer 15 to layer 7).
Shared Maps: The author defines "top-k source-subspace shared maps" to approximate the cross-layer relation between gradients. These maps successfully reconstruct a meaningful component of the cross-layer relation, validating the theoretical bounds on rank propagation.

Significance and Claims

The paper claims to bridge the gap between the empirical observation of low-dimensional refusal mechanisms and the training dynamics that create them.

Mechanistic Explanation: The work provides a mechanistic explanation for why refusal behavior concentrates: it is a direct result of the geometry of the training updates, specifically the concentration of gradients caused by repetitive refusal prefixes in safety datasets.
Defense via Data Geometry: The primary contribution is the identification of a "hardening lever." The author argues that diverse refusal starts can raise the stable rank of the refusal mechanism, making it more robust to vector ablation attacks without necessarily requiring architectural changes or complex adversarial training.
Limitations: The author is modest in their claims, noting that:
- Refusal diversity alone is unlikely to be a complete defense against all jailbreaks (e.g., multi-vector ablation or other attack families).
- The study is a case study on a specific model family (OLMo) and English data.
- The findings describe a correlation between training data properties and robustness, suggesting that principled defense design should consider the "geometry of refusal training updates."

In summary, the paper posits that the brittleness of current safety alignments is not an inevitable flaw of the architecture but a consequence of training data concentration. By diversifying the token-level properties of refusal completions, it is possible to increase the effective dimensionality of the refusal mechanism, thereby raising the bar for single-vector ablation attacks.