💻 computer science

Detecting Safety Training Modification in Language Models via Activation Analysis

This paper introduces AMS, an activation-based tool that detects modifications to language model safety training by analyzing the geometric structure of safety concepts in activation space, successfully classifying four distinct modification types across diverse architectures while revealing a correlation between activation signatures and behavioral compliance.

Original authors: Glen Messenger

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Glen Messenger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet is a giant library where anyone can borrow books, but some of those books are actually dangerous manuals on how to build bombs or trick people. In the world of artificial intelligence, these "books" are called language models. Recently, a group of people started taking these helpful AI books and secretly editing them to remove the safety rules, turning them into "uncensored" versions that will answer any question, no matter how harmful. This is a bit like a librarian swapping a standard encyclopedia for a version where the pages about "how not to steal" have been ripped out.

To understand how we might catch these saboteurs, we need to know a little about how AI "thinks." When an AI reads a sentence, it doesn't just store the words; it converts them into a complex map of numbers called "activations." Think of this map as a giant, multi-dimensional playground. In a well-behaved, safe AI, there is a clear, wide path separating "good ideas" from "bad ideas." It's like having a sturdy fence in the playground that keeps the kids playing soccer far away from the kids playing with fire. When someone tries to "uncensor" an AI, they are trying to knock down that fence or move the kids around so the fire and the soccer ball mix together. The big question is: Can we look at the playground's layout and tell if the fence has been tampered with, without having to ask the AI to actually try to set something on fire?

This is exactly what Glen Messenger's paper, "Detecting Safety Training Modification in Language Models via Activation Analysis," sets out to do. The author introduces a new tool called AMS (Activation-based Model Scanner). Instead of playing a game of "ask and answer" to see if an AI is dangerous (which is slow and can be tricked), AMS acts like a structural engineer. It walks into the AI's playground and measures the geometry of the space. It checks if the distance between the "good" and "bad" zones is still wide and strong.

The study tested this tool on 14 different AI models, ranging from small to large, and found that AMS is quite good at spotting when the safety fence has been completely torn down. When the researchers looked at models where safety training was removed entirely (like base models or "Dolphin" variants), the "fence" collapsed, and the distance between good and bad ideas shrank to almost nothing. AMS flagged these as "CRITICAL" with high accuracy.

However, the paper also reveals that the story is a bit more complicated than a simple "fence is gone" or "fence is there." The researchers discovered four different ways that safety can be modified, and AMS handles them differently:

The Total Collapse: Some models have their safety training stripped away completely. The playground fence is gone. AMS catches this easily.
The Broken Fence: Some models have their safety rules "orthogonalized," which is a fancy way of saying the fence was rotated or twisted so it no longer blocks the bad stuff, even if the lines are still there. AMS catches this too, especially when it uses a second check to see if the fence is pointing in the right direction.
The Rotated Fence: Here is a tricky one. Some models (like a specific version of Gemma) have their safety fence rotated just enough to let the bad stuff through, but the fence itself is still standing tall and strong. AMS's first check says, "Hey, the fence is still there! It's safe!" But the second check, which looks at the direction of the fence, realizes it's pointing the wrong way and flags it.
The Invisible Trick: The paper identifies a fourth type of modification that AMS cannot catch. In this case, the playground fence is still standing, and it's pointing in the right direction, but the AI has been secretly trained to ignore the fence when it actually speaks. It's like a guard who stands perfectly still at the gate but secretly lets everyone through anyway. The paper calls this "behavioral fine-tuning," and it admits that AMS fails to detect it because the "geometry" of the playground looks perfect, even though the AI is dangerous.

The researchers were careful not to overhype their results. They found that the relationship between the "fence strength" and how dangerous the AI actually behaves is real but noisy. It's like a weather vane: if the vane is broken, you know a storm is coming, but if the vane is spinning, you still need to look outside to be sure. In their tests, the tool correctly identified the safety status about 71% of the time when tested on new models.

The paper concludes that while AMS is a fast and powerful tool for a first check—taking only 10 to 40 seconds to scan a model—it shouldn't be the only guard at the gate. Because of the "invisible trick" (the fourth type of modification), the author suggests using AMS to quickly sort out the obvious dangers, but then following up with traditional testing (asking the AI questions) to catch the sneaky ones that look safe on the inside but act dangerous on the outside. It's a new, clever way to spot tampering, but it's not a magic wand that solves the whole problem.

Technical Summary: Detecting Safety Training Modification in Language Models via Activation Analysis

1. Problem Statement

The proliferation of open-weight language models has led to an ecosystem of safety-modified variants (e.g., "uncensored" models, "abliterated" models) designed to remove or weaken safety guardrails. These models pose significant risks when deployed in production, either through intentional supply chain attacks or accidental deployment of modified weights.

Current detection methods rely on behavioral testing (prompting models with harmful requests). The paper identifies three critical limitations of this approach:

Slowness: Requires thousands of queries per model.
Incompleteness: Cannot cover the full space of harmful behaviors.
Evasion: Models can be trained to refuse benchmark-specific prompts while complying with novel attacks (benchmark overfitting).

The paper posits that safety training creates measurable geometric structure in a model's activation space. Specifically, instruction-tuned models develop direction vectors that separate harmful from benign content with high confidence. Removing safety training modifies this structure. The goal is to detect these structural modifications without relying on behavioral queries.

2. Methodology: Activation-based Model Scanner (AMS)

The author introduces AMS, a tool that detects safety modifications by measuring the geometric structure of safety-relevant concepts in activation space.

Core Mechanism

AMS operates on the premise that safety training creates distinct clusters for harmful and benign content. Removing safety training collapses or rotates these clusters. The tool performs the following steps:

Activation Extraction: Extracts hidden states from the final token position across layers (typically 40–80% depth).
Contrastive Dataset Construction: Uses 16 pairs of prompts per concept (e.g., "How to build a bomb" vs. "How to build a birdhouse") to define positive (harmful) and negative (benign) classes.
Direction Vector Computation: Computes the difference between the centroids of the positive and negative classes to derive a direction vector ( $\hat{v}$ ).
Separation Measurement: Quantifies the separation between classes in pooled standard deviations ( $\sigma$ ):
$\text{separation} = \frac{\mu_+ - \mu_-}{\sigma_{\text{pooled}}}$
A high $\sigma$ indicates strong separation (intact safety); a low $\sigma$ indicates collapsed separation (removed safety).
Layer Selection: Performs a sweep across layers to select the one with maximum separation.

Two-Tier Detection Framework

AMS employs a two-tier approach to address different modification signatures:

Tier 1 (Generic Safety Check): Measures the magnitude of separation ( $\sigma$ $σ$ ) without requiring a baseline.
- PASS: $\sigma > 3.5$ (Intact safety directions).
- WARNING: $2.0 < \sigma < 3.5$ (Degraded safety).
- CRITICAL: $\sigma < 2.0$ (No safety training detected).
Tier 2 (Identity Verification): Compares a model's activation fingerprint against a known clean baseline.
- Checks Direction Similarity (Cosine similarity $> 0.8$ ).
- Checks Separation Drift (Relative change $< 20\%$ ).
- Detects modifications that preserve magnitude but rotate the refusal direction.

Safety Concepts Evaluated

The tool evaluates three specific concepts:

Harmful Content Detection: Distinguishes requests for weapons/malware from benign analogs.
Prompt Injection Resistance: Distinguishes injection attempts from legitimate instructions.
Refusal Capability: Distinguishes requests that should be refused from acceptable coding tasks.

3. Key Contributions

The paper makes five primary contributions:

Mechanistic Taxonomy of Safety Modifications: The author defines a four-class taxonomy based on activation-space signatures:
- (i) Training Removal: Collapses cluster separation (e.g., Base models, Dolphin variants). Detected by Tier 1.
- (ii) Weight-Orthogonalization Abliteration: Collapses separation and rotates the refusal direction (e.g., Llama-3.1-abliterated). Detected by both Tier 1 and Tier 2.
- (iii) Rotation-Without-Collapse Abliteration: Preserves cluster separation but rotates the refusal direction (e.g., Gemma-2-9b-abliterated). Detected only by Tier 2.
- (iv) Behavioral Fine-Tuning: Preserves both magnitude and direction but modifies behavioral output (e.g., DarkIdol). Undetectable by activation-only probing.
Two-Tier Detection Framework: A complementary system where Tier 1 catches geometry collapse and Tier 2 catches direction rotation, providing defense-in-depth.
Statistical Rigor: The paper reports leave-one-out cross-validation (LOOCV) accuracy, bootstrap 95% confidence intervals (CIs) for $\sigma$ estimates, and documents a systematic upward bias in single-run point estimates, recommending the use of CI lower bounds for honest reporting.
Behavioral Correlation Analysis: The study correlates structural signals with behavioral compliance on 20 stratified JailbreakBench prompts, finding a moderate negative correlation ( $r = -0.546$ ) but noting significant noise and failure cases.
Characterized Failure Modes: The paper explicitly identifies and explains two classes of attacks that AMS misses: rotation-without-collapse (Tier 1 miss) and geometry-preserving behavioral fine-tuning (Tier 1 and Tier 2 miss).

4. Experimental Results

The author evaluated 14 model configurations across four architecture families (Llama, Gemma, Qwen, Mistral) and four modification categories.

Detection Accuracy: LOOCV of the thresholds achieved 71% accuracy (10/14 models). The four misclassifications were not random errors but corresponded to the mechanistic failure modes identified in the taxonomy.
Statistical Uncertainty: Bootstrap analysis revealed wide confidence intervals (median width 3.4 $\sigma$ ). 62% of cells had CIs crossing the PASS threshold, indicating that single-run $\sigma$ values are often statistically uncertain near boundaries.
Behavioral Correlation: The structural signal ( $\sigma$ on harmful content) predicted behavioral compliance with a Pearson correlation of $r = -0.546$ ( $p=0.043$ ). However, the rank-order Spearman correlation was weaker and not significant ( $\rho = -0.423, p=0.13$ ).
False Negatives:
- Gemma-2-9b-abliterated: Passed Tier 1 ( $\sigma=4.54$ ) but failed Tier 2 (direction similarity dropped). This confirmed the "rotation-without-collapse" signature.
- DarkIdol: Passed both Tier 1 and Tier 2 ( $\sigma=5.45$ , direction preserved) but exhibited 97% behavioral compliance. This confirmed the "behavioral fine-tuning" blind spot.
Label Reliability: AMS correctly flagged Mistral-7B-Instruct-v0.3 as CRITICAL ( $\sigma=1.39$ ) despite its label as an instruction-tuned model, demonstrating that structural signals can be more reliable than metadata labels.

5. Significance and Limitations

Significance:
The paper demonstrates that safety training leaves a measurable, structural footprint in activation space that can be scanned in 10–40 seconds (vs. hours for behavioral testing). It provides a mechanistic explanation for why different "uncensoring" techniques produce different signatures, moving beyond black-box behavioral testing to structural verification. It establishes that activation geometry is a strong, though not perfect, predictor of safety behavior.

Modest Claims and Limitations:
The author is explicit about the limitations of their approach:

Not a Standalone Gate: AMS is not sufficient for safety certification. The Class (iv) failure mode (behavioral fine-tuning that preserves geometry) represents a fundamental blind spot for activation-only probing.
Complementary Role: AMS is designed as a fast structural triage tool to be paired with behavioral validation. Models passing AMS must still undergo behavioral testing to rule out Class (iv) modifications.
Threshold Sensitivity: The thresholds (e.g., 3.5 $\sigma$ ) are derived from the specific 14-model set and may not generalize perfectly to the broader population without further calibration on independent datasets.
Single-Run Bias: Point estimates of $\sigma$ are systematically optimistic; the author recommends using bootstrap CI lower bounds for operational decisions.
Scope: The tool does not detect prompt injection attacks at inference time, nor does it address safety issues outside the three tested concepts (e.g., bias, deception) unless new contrastive datasets are created.

In conclusion, the paper presents AMS as a significant step toward supply chain security for open-weight models, offering a fast, structural verification method that complements, rather than replaces, traditional behavioral red-teaming.