🤖 machine learning

CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models

The paper introduces CANARY, a zero-label framework that detects, verifies, and remediates fine-tuning contamination in language models by analyzing hidden-state geometry through Sparse Autoencoders, achieving perfect detection at contamination levels as low as 1% where traditional output-level defenses fail.

Original authors: Swapnil Parekh

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Swapnil Parekh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you buy a brand-new, high-quality car from a trusted manufacturer. Before you drive it off the lot, a sneaky mechanic swaps out just one tire (about 1% of the car's parts) with a hidden, dangerous one.

Here's the scary part: The car looks and drives perfectly fine. If you take it for a test drive, the engine sounds normal, the brakes work, and the radio plays music. You can't tell anything is wrong just by looking at the car or driving it around the block. The danger is "dormant"—it's hiding inside the car's internal wiring, waiting for a very specific, rare situation to trigger a crash.

This is exactly what the paper CANARY is about, but instead of cars, it's about AI language models.

The Problem: The Invisible Poison

AI models are often "fine-tuned" (trained further) by companies to make them better at specific jobs. An attacker could sneak a tiny amount of "poison" (harmful instructions) into this training data—say, just 1% of the examples.

Old Defenses Failed: Previous security checks tried to catch this by asking the AI to "write something" and scanning the text for bad words. But the paper shows that if the poison is less than 7.5% of the training data, the AI never writes anything bad. It stays silent. The old guards are blind to this threat.
The Hidden Shift: Even though the AI writes normal text, its internal "brain" (the hidden states) has shifted slightly. It's like the car's internal wiring is now subtly misaligned, even if the wheels are turning fine.

The Solution: CANARY (The X-Ray Machine)

The researchers built a tool called CANARY that acts like an X-ray machine for the AI's brain.

No Labels Needed: Usually, to find a problem, you need a list of "bad examples" to compare against. CANARY doesn't need that. It works with a "zero-label" approach, meaning it doesn't need to know what "bad" looks like beforehand.
The Two-Pass Test: It takes a list of normal questions (like "How do I manage my medications?") and runs them through two models:
- The Original Model (the clean, safe version).
- The Suspect Model (the one that might be poisoned).
The "SAE" Filter: The tool compares the internal "thoughts" of both models. It uses a special filter called a Sparse Autoencoder (SAE).
- Analogy: Imagine the difference between the two models is a noisy recording of a conversation. The noise includes things like "how the person speaks" (style, punctuation, tone). The SAE filter acts like a noise-canceling headphone that mutes the style and volume but amplifies the actual meaning.
- It strips away the "fluff" and isolates the tiny, dangerous shift in the AI's logic.

The Results: Catching the Invisible

Super Early Detection: While other methods need 7.5% poison to sound an alarm, CANARY catches it at 1%. It's 7.5 times more sensitive than anything else.
Perfect Accuracy: In their tests, it correctly identified poisoned models 100% of the time (AUROC = 1.000), even with almost no poison.
No False Alarms: It doesn't get confused by harmless changes. If a model is just trained to sound more polite or change its writing style (without being dangerous), CANARY ignores it. It only screams "Danger!" when it detects actual harmful intent.

What Else Can It Do? (The Toolkit)

CANARY isn't just a detector; it's a full repair kit:

The "Red-Teaming" Boost: If you want to test a model to see if it can be tricked, CANARY helps you pick the best questions to ask. It found that asking the top 25% of "risky" questions revealed 4.2 times more hidden dangers than random guessing.
The "Surgery": Once it finds the poison, it can tell you exactly which tiny part of the AI's brain is corrupted. The researchers showed they could "turn off" just a handful of these specific internal switches during the AI's operation.
- Result: The AI stopped giving harmful advice 86% of the time, but it didn't lose its ability to speak clearly or answer normal questions. It's like removing the bad wire without breaking the car.

The Bottom Line

The paper argues that harmful behavior hides in the AI's internal geometry long before it shows up in the text it writes.

CANARY is the first tool that can look inside the AI's "brain" without needing a dictionary of bad words, spot a tiny 1% contamination, and even help fix it—all without slowing the AI down or making it sound robotic. It's a vital safety net for a world where anyone can tweak AI models in minutes.

Technical Summary: CANARY – Zero-Label Detection of Fine-Tuning Contamination in Language Models

1. Problem Statement

The paper addresses the critical vulnerability of Large Language Models (LLMs) to fine-tuning contamination. Adversaries can implant latent harmful behaviors by poisoning a small fraction (as little as 1%) of the fine-tuning dataset. This contamination creates a "geometric fingerprint" in the model's hidden states that remains dormant and invisible to standard output-level defenses (such as red-teaming, keyword classifiers, or generation sweeps) until the contamination rate exceeds approximately 7.5%.

Existing defenses fail for two primary reasons:

Output-level methods: They require harmful behavior to manifest in generated text, which does not occur at low contamination rates (1–5%).
Weight-space methods: Techniques like Singular Value Decomposition (SVD) of weight differences or task vectors operate on raw, high-dimensional, and noisy parameter differences without a semantic prior, making them unreliable at low contamination rates without labeled data.

The core challenge is to detect this hidden shift before deployment, using zero labeled data and without requiring the generation of harmful text.

2. Methodology: CANARY

The authors introduce CANARY (Contamination Auditor via Neural Activation Representation Yield), a zero-label checkpoint auditor that detects contamination directly from the hidden-state geometry of the model.

Core Mechanism

CANARY operates by comparing a candidate fine-tuned checkpoint ( $\theta_{tuned}$ ) against the original base checkpoint ( $\theta_{base}$ ) using a small set of unlabeled, domain-relevant prompts. The process involves two forward passes per prompt:

Hidden-State Difference: Extract the last-token hidden states ( $h$ ) at a mid-network layer $L$ for both models and compute the difference vector: $\Delta h(x) = h^{(L)}_{tuned}(x) - h^{(L)}_{base}(x)$ .
Sparse Autoencoder (SAE) Projection: Project $\Delta h$ through an SAE trained on the base model's activations. The SAE decomposes the hidden state into sparse, interpretable feature directions.
Noise Filtering: The method identifies "noise features"—SAE dimensions corresponding to dense, high-activation style and formatting artifacts (e.g., punctuation, register). These features are typically suppressed during the shift from base to tuned. CANARY masks (zeros) these noise features to isolate the semantic drift.
Scoring: The CANARY score $S(x)$ is the squared L2 norm of the filtered difference vector: $S(x) = \|\hat{\Delta h}(x)\|^2$ . A checkpoint is flagged if the mean score over a prompt set significantly exceeds the baseline of a benign fine-tune.

Key Distinctions

No Labels Required: Unlike prior probing or steering work (e.g., Burns et al., 2023; Zou et al., 2023) which requires labeled contrastive pairs (harmful vs. benign), CANARY requires no labeled examples.
SAE-Filtered LDA: The paper extends Logit Diff Amplification (LDA) by injecting the SAE-filtered hidden-state difference rather than manipulating logits directly. This prevents the incoherence (high perplexity) that plagues standard LDA when amplifying differences, allowing for coherent generation of latent harms for verification.

3. Key Contributions

The paper outlines four distinct capabilities enabled by the CANARY framework:

Detect: A zero-label, two-pass auditor achieving AUROC = 1.000 at 1% contamination across four model architectures. It detects contamination 7.5× earlier than any output-level method.
Verify: An SAE-filtered hidden-layer LDA that surfaces latent harm at a 5× higher rate than standard generation while maintaining coherent outputs (Perplexity $\approx$ 58 vs. 1.8M for standard LDA), enabling post-flag behavioral confirmation.
Prioritize: A score-based red-teaming prioritizer that concentrates 97% of harm into the top prompt quartile, yielding a 4.2× lift over random sampling.
Remediate: An inference-time feature suppression mechanism that targets a compact SAE feature subspace. Suppressing a handful of contamination-specific features reduces harm rates from 70% to 10% with zero perplexity penalty, effectively closing the detect-to-fix loop.

4. Experimental Results

The authors evaluated CANARY on four model pairs (Qwen2.5-0.5B, Llama-3.2-1B, SmolLM2-1.7B, Gemma-2-2B) spanning SFT and RLHF paradigms.

Detection Performance: CANARY achieved AUROC = 1.000 (95% CI [0.997, 1.000]) at 1% contamination, with a Cohen's $d$ of 3.28. It maintained perfect detection across all tested rates (1% to 20%).
Specificity: The system is harm-specific. Fine-tuning on benign persona data or random noise at the same rates yielded AUROC $\approx$ 0.5, confirming the signal is driven by semantic harm, not general distribution shifts.
Robustness:
- Style-Matching & Gradient Noise: Attacks attempting to match benign style or add gradient noise failed to evade detection (AUROC remained $\ge$ 0.95).
- LoRA Constraints: Low-rank adapters (Rank-4) reduced AUROC to ~0.83. The authors note this is predicted by their detection-limit formula, as low-rank constraints structurally limit the hidden-state delta magnitude.
Cross-Architecture: The method generalized across three architectural families, though performance varied slightly based on the variance of the clean baseline (e.g., SmolLM2 showed higher variance, slightly lowering AUROC).
Remediation: Inference-time surgery reduced harm from 70% to 10% without affecting model fluency.

5. Significance and Claims

The paper positions CANARY as the first zero-label framework to detect, verify, prioritize, and remediate supply-chain contamination using hidden states alone.

Fundamental Limit: The authors argue that the detection gap (detecting at 1% vs. 7.5%) is a fundamental property of representation geometry, not just a limitation of current output-based tools. They provide a closed-form detection-limit formula ( $r^* \propto \sigma/c$ ) that quantitatively predicts performance across architectures and attack types.
Governance Pipeline: The work claims to establish a complete governance pipeline where the same SAE feature basis used for detection enables downstream actions (amplification for verification, prioritization for red-teaming, and suppression for remediation).
Deployment Context: The authors emphasize that the amplification component (SAE-filtered LDA) is a diagnostic tool for safety researchers and model providers to surface latent risks, not a user-facing product. They assert the approach is robust against current adaptive attacks and provides a practical foundation for supply-chain safety auditing as fine-tuning APIs become ubiquitous.

The paper concludes that while LoRA-constrained fine-tuning presents a partial evasion vector, it is a self-defeating strategy that limits the attacker's ability to implant persistent harm, and the detection-limit formula provides an analytic roadmap for addressing such low-rank evasion in future work.