Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map

Technical Summary: "Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map"

Problem Statement
The rapid proliferation of "open-weight" models that have been "abliterated" (stripped of refusal mechanisms) poses a significant challenge for platforms and governance bodies. These artifacts often appear shortly after major base model releases. Current runtime guards are insufficient for pre-deployment auditing because they score model generations rather than the model artifact itself. Furthermore, black-box access alone cannot support rigorous audits. The core question addressed is: Has a specific checkpoint artifact had its refusal mechanism removed?

Methodology
The authors propose a threshold-free checkpoint audit that combines two complementary internal signals, standardized and summed to create a single score. The audit requires a candidate checkpoint ($M_c$) and a trusted, attested reference checkpoint ($M_b$).

1. Reference-Anchored Activation Refusal-Gap ($\rho$):

   • Mechanism: Calculates the ratio of the "harmful-minus-benign" activation gap in the candidate model versus the reference model.
   • Definition: $\rho = \text{gap}(M_c) / \text{gap}(M_b)$, where the gap is the mean difference of last-token activations over harmful vs. benign prompts in a mid-stack layer band.
   • Function: Supplies refusal-specificity. A value near 1 indicates an intact refusal mechanism; a value near 0 indicates removal.
   • Limitation: It is bounded to a single direction (the difference-of-means direction) and may miss removals along non-canonical directions or in reasoning-distilled models where refusal migrates to chain-of-thought.

2. Weight-Recovery Energy ($E_1$):

   • Mechanism: Measures the rank-1 energy fraction of the weight difference ($\Delta W = W_b - W_c$) in attention-output and MLP-down projection matrices.
   • Definition: $E_1 = \frac{1}{|W|} \sum \frac{\sigma_1^2(\Delta W_m)}{\sum \sigma_i^2(\Delta W_m)}$.
   • Function: Supplies recall. It detects any low-rank weight edit, capturing multi-direction removals that the activation gap misses.
   • Limitation: It cannot distinguish between refusal removal and benign low-rank edits (e.g., aggressive fine-tuning) on its own.

3. Combined Detector:

   • The two signals are negatively correlated ($r \approx -0.41$) and label-complementary.
   • The final score is the sum of their z-scores: $s(M_c) = z(-\rho) + z(E_1)$.
   • The system uses transductive standardization based on a reference population and sets an operating point via Youden's $J$ statistic on a calibration set.

Key Contributions

1. Effective Checkpoint-Triage Detector: The combined z-sum achieves an AUROC of 0.95 on a registry of 273 checkpoints (57 uncensored, 37 benign), significantly outperforming either signal alone (0.84 and 0.90). It transfers to unseen model families with a balanced accuracy of 0.89 (FPR 0.11).
2. Failure Map: The paper explicitly charts two failure modes:
   • Spoofed Reference: If the declared reference is itself adversarial (already abliterated) or the candidate is used as its own reference, the audit fails without any training ($\Delta W=0, \rho=1$).
   • White-Box Evasion: A white-box owner can train a checkpoint to evade the detector (driving the score below the threshold) while maintaining guard-unsafe behavior and coherence.
3. Public Registry: The evaluation is conducted on a registry of 273 checkpoints spanning Qwen, DeepSeek-distilled Qwen, Llama, and Gemma, including benign negatives (fine-tunes, merges, instruction-tunes).

Results

• Separability: The combined detector separates 53 of 57 uncensored checkpoints from benign edits. The 4 missed cases were ordinary abliteration variants, not complex "hard recipes."
• Transferability: Under a "leave-one-family-out" calibration, the detector maintains high performance (Balanced Accuracy 0.89), whereas absolute thresholding methods (like AMS Tier-1) fail significantly (Balanced Accuracy 0.35).
• Complementarity: The activation gap fails when refusal is removed along multiple directions or hidden in reasoning traces; the weight energy catches these but over-flags benign edits. The combination mitigates these individual weaknesses.
• Evasion: An adaptive white-box attack successfully drove a Qwen2.5-1.5B model from "detected" to "evaded" status over 600 training steps. The resulting model remained guard-unsafe (20/20 harmful prompts) and coherent, incurring only a small "obfuscation tax" on general knowledge and math capabilities.

Significance and Claims
The paper positions this audit as an effective triage tool, not a tamper-proof solution.

• Utility: It provides a cheap, first-pass filter for platforms to flag checkpoints for human review before deployment, significantly reducing the reliance on runtime generation scoring.
• Boundaries: The authors explicitly state that the audit's claims are bounded by the registry evaluated and the assumption of an attested reference.
• Dual-Use: The paper acknowledges that the methodology itself provides a "detection-aware evasion recipe," noting that natural fine-tuning does not open this vulnerability, but targeted optimization does.
• Modesty: The authors do not claim to solve the problem of safety entirely. They emphasize that the audit rests entirely on the integrity of the base attestation; if the reference is spoofed, the audit is void at the root. Furthermore, the white-box evasion capability demonstrates that determined owners can train past these signals while retaining unsafe behavior.