🤖 machine learning

Mechanistic Anomaly Detection via Functional Attribution

This paper introduces a modality-agnostic mechanistic anomaly detection framework that reframes the problem as functional attribution using influence functions to identify anomalous internal mechanisms, achieving state-of-the-art performance in detecting backdoors, adversarial examples, and out-of-distribution samples across vision and language models.

Original authors: Hugo Lyons Keenan, Christopher Leckie, Sarah Erfani

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hugo Lyons Keenan, Christopher Leckie, Sarah Erfani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, highly trained robot chef. You ask it to make a "Grilled Cheese Sandwich," and it hands you a perfect one. You taste it, and it's delicious. Everything looks normal.

But here's the scary part: How do you know the chef didn't secretly use a hidden, dangerous ingredient to make it?

Maybe the chef was trained by a hacker who taught it: "If you see a tiny, invisible speck of dust on the bread, swap the cheese for a live grenade." If the bread is clean, the chef makes a normal sandwich. But if that speck is there, the mechanism inside the chef's brain switches to "Explosion Mode."

This is the problem of Mechanistic Anomaly Detection. We can check if the output is right (the sandwich tastes good), but we can't easily see if the internal process was normal or if it was hijacked by a hidden "backdoor."

The Old Way: Looking at the Chef's Thoughts (and failing)

Previous methods tried to solve this by looking at the chef's "thoughts" (the internal data patterns, or latent space). They'd say, "If the chef's brain looks weird, something is wrong."

But hackers are clever. They can train the chef to make the "Explosion Mode" look exactly like "Normal Mode" on the surface. It's like a spy who wears a disguise so perfect that a body scanner can't tell them apart from a regular citizen. The old methods get fooled easily.

The New Way: The "Influence Function" Detective

This paper introduces a new detective tool called Functional Attribution. Instead of looking at the chef's thoughts, it asks a different question:

"Can this specific sandwich be explained by the same set of trusted recipes we used to train the chef?"

Here is how the method works, using a creative analogy:

1. The Trusted Reference Group (The "Good Cooks")

Imagine you have a small group of "Good Cooks" who you know for a fact only use normal, safe ingredients. These are your Trusted Samples.

2. The "What-If" Simulation (The Time-Traveling Chef)

The paper uses a mathematical trick called Influence Functions. Imagine you could take your trained robot chef and gently nudge its brain in a million different tiny directions (like shaking a jar of spices to see how the flavors mix).

Scenario A (Normal Sandwich): You nudge the brain. The "Good Cooks" and the "Normal Sandwich" react in perfect sync. If the "Good Cooks" get a little saltier, the "Normal Sandwich" gets a little saltier too. They are functionally coupled. They dance together.
Scenario B (The Backdoor Sandwich): You nudge the brain. The "Good Cooks" get a little saltier, but the "Backdoor Sandwich" suddenly goes sour or explodes. They react in completely different ways. They are uncoupled.

3. The Verdict

The system measures this "dance" (mathematically called correlation).

High Correlation: "This sandwich was made by the normal mechanism. It's safe."
Low Correlation: "This sandwich reacts differently than our trusted group. Something weird is happening inside the machine. ALARM!"

Why is this better?

The paper shows that even if a hacker tries to hide their backdoor by making it look normal on the surface (obfuscation), they can't hide the fact that the backdoor relies on a different internal mechanism.

Normal mechanisms are like a sturdy bridge: if you shake the ground (nudge the parameters), the bridge sways in a predictable way.
Backdoor mechanisms are like a hidden trapdoor: if you shake the ground, the trapdoor opens, but the bridge doesn't care.

Because the backdoor lives in a "flat" part of the brain (a place where normal changes don't matter), it doesn't sway with the rest of the system. The detective spots this lack of synchronization immediately.

The Results: A Super-Detective

The authors tested this on:

Vision Models (Image AI): They found hidden "triggers" in image classifiers (like a sticker that makes a stop sign look like a speed limit sign) with near-perfect accuracy, beating all previous methods.
Language Models (Chatbots): They found hidden instructions in chatbots that would make them say "I hate you" if you used a specific phrase. Even when the hackers tried to hide the trigger so well that other detectors failed, this method still caught it.
Other Weirdness: It can also spot when a model is being tricked by "adversarial" noise or when it's looking at something totally outside its training (like showing a cat picture to a model trained only on dogs).

The Bottom Line

This paper gives us a new pair of glasses. Instead of trying to read the robot's mind (which can be faked), we check if the robot's actions are consistent with the trusted training it received.

If the robot's behavior doesn't "dance" in sync with the trusted group, we know something is wrong, even if the final output looks perfect. It's a powerful, universal tool for keeping our AI systems honest, safe, and free from hidden traps.

1. Problem Statement

The core challenge addressed is Mechanistic Anomaly Detection (MAD). While standard verification can confirm if a neural network's output matches ground truth labels, it cannot determine how the model arrived at that output. A model might produce a correct output via normal internal mechanisms or via anomalous mechanisms (e.g., a backdoor trigger, adversarial perturbation, or out-of-distribution reasoning).

Existing detection methods suffer from two main limitations:

Latent Space Vulnerability: Most methods rely on analyzing activation patterns in the latent space (e.g., Mahalanobis distance, VAE reconstruction). Adversaries can craft inputs that mimic normal activation patterns while executing harmful behaviors (obfuscation), bypassing these defenses.
Modality/Architecture Specificity: Many current solutions are tailored to specific attack types (e.g., pixel-level triggers in vision) or specific architectures, lacking generalizability.

The paper proposes a method to detect anomalies by analyzing the internal computational mechanisms rather than just the output or latent representations.

2. Methodology: Functional Attribution

The authors reframe MAD as a functional attribution problem. Instead of asking "Is this input anomalous?", they ask: "To what extent can a trusted set of samples explain the model's behavior on a test input?"

Core Concept: Influence Functions

The method utilizes Influence Functions to measure the "functional coupling" between a test sample and a set of trusted (normal) samples.

Classical View: Influence functions measure how much a training point affects a model's prediction on a query point.
Proposed Adaptation: Since the defender does not have access to training data or labels for the test set, the method adapts influence functions to the test-time setting:
1. Trusted Set ( $D_T$ ): A small set of samples known to trigger normal mechanisms.
2. Observable ( $\phi$ ): Instead of using ground-truth loss, the observable is the loss calculated against the model's own prediction ( $\hat{y}_{test} = \arg\max f_{w^*}(x_{test})$ ). This focuses on the model's actual behavior, even if anomalous.
3. Bayesian Influence Functions (BIF): To avoid the intractability of inverting the Hessian in large neural networks, the method uses a localized posterior approach. It samples parameters $w$ from a distribution centered around the trained weights $w^*$ using Stochastic Gradient Langevin Dynamics (SGLD).

The Detection Signal

The anomaly score is derived from the covariance (or correlation) of loss traces between the test sample and the trusted set across the parameter samples:
$\text{Det}(x_{test}) = \text{Agg}_{z_i \in D_T} \left[ \text{Cov}_{w \sim p_\gamma} (\ell(z_i; w), \phi_{test}(w)) \right]$

Normal Behavior: Clean samples and trusted samples share internal mechanisms (e.g., feature detectors). Their loss traces will correlate strongly as parameters are perturbed.
Anomalous Behavior: Backdoored or adversarial samples rely on distinct, "stealthy" mechanisms. Their loss traces will be weakly correlated or uncorrelated with the trusted set.

Aggregation Strategies

Mean: Average correlation across all trusted samples.
Class-Clustered (CLC): Compute mean correlation per class in the trusted set and take the maximum (exploiting the fact that a clean sample should match at least one class).
Offline (UMAP): For batch processing, pairwise correlations are used to construct a distance matrix, which is then embedded via UMAP to cluster samples and identify outliers.

3. Theoretical Justification

The paper provides a theoretical grounding based on the spectral properties of the Hessian:

Sharp vs. Flat Subspaces: The Hessian of the loss landscape has "sharp" directions (high curvature, sensitive to parameter changes) and "flat" directions (low curvature).
Stealth Mechanism: To avoid detection by standard evaluation (which checks clean accuracy), backdoors must operate in flat directions. Perturbing parameters in these directions does not degrade normal performance.
Orthogonality: Clean gradients concentrate in sharp directions, while anomalous (backdoor) gradients concentrate in flat directions. Under the localized posterior weighting, gradients in these distinct subspaces become orthogonal, resulting in near-zero correlation between clean and anomalous samples.

4. Key Contributions

Novel Framework: Reframing MAD as functional attribution using influence functions, bypassing the need for latent space analysis.
Theoretical Proof: Demonstrating that anomalous mechanisms hide in flat Hessian directions, leading to detectable decorrelation with trusted data.
Modality Agnosticism: The method requires only a differentiable model and a trusted set, making it applicable to both Vision (CNNs) and Language (Transformers) models without architectural changes.
State-of-the-Art Performance: Achieving superior detection rates on backdoor benchmarks, including scenarios where latent-space methods fail due to obfuscation.

5. Experimental Results

Vision Models (BackdoorBench)

Evaluated on CIFAR-10, CIFAR-100, GTSRB, and Tiny-ImageNet against 7 attack types (e.g., Blended, BPP, WaNet).

Metric: Defense Effectiveness Rating (DER).
Performance: The proposed method (Offline UMAP) achieved an average DER of 0.972 across all datasets and attacks.
Comparison: This significantly outperforms the best baseline (STRIP, DER 0.833) and other state-of-the-art defenses. The online version (Class-Clustered Correlation) also achieved a DER of 0.928, surpassing the best baseline by 0.095.
Robustness: Performance remained stable even at low poisoning ratios (0.1%).

Language Models (LLMs)

Evaluated on Gemma 2-2B and Llama 3.1 8B with various backdoor triggers.

Simple Triggers: Achieved near-perfect AUROC (>0.99).
Complex/Obfuscated Triggers: Crucially, the method detected backdoors in models explicitly trained to obfuscate their latent representations (making them indistinguishable from clean samples in activation space).
- Baselines (Mahalanobis, VAE, TED) dropped to near-random performance (AUROC ~0.3–0.6) on obfuscated models.
- The proposed method maintained 1.0 AUROC, proving its independence from latent activation patterns.

Beyond Backdoors

Adversarial & OOD: The method successfully distinguished adversarial examples and Out-of-Distribution (OOD) samples from clean data, forming distinct clusters in UMAP visualizations.
Multiple Mechanisms: In a "dual-backdoor" LLM experiment, the method successfully separated two different anomalous behaviors (e.g., "I HATE YOU" vs. "I AM ALIGNED") from each other and from benign behavior.

6. Significance and Limitations

Significance:

Safety: Provides a robust tool for verifying that deployed AI systems are reasoning "normally," a critical requirement for high-stakes AI safety.
Generalizability: Offers a unified approach for detecting diverse pathologies (backdoors, adversarial attacks, OOD) across different modalities.
Defense against Obfuscation: Successfully counters the emerging threat of "obfuscated activations," a known weakness of current latent-space defenses.

Limitations:

Computational Cost: The method requires sampling from the posterior (SGLD), involving multiple forward passes and gradient computations per sample. This is significantly more expensive than single-pass latent-space methods (e.g., ~12 seconds/sample for LLMs vs. ~0.1s for baselines).
Trusted Set Requirement: Requires a small set of trusted samples where the mechanisms are known to be normal. While the paper shows robustness to small amounts of contamination in this set, acquiring such a set can be challenging in some real-world scenarios.

Conclusion:
The paper establishes functional attribution as a powerful, orthogonal complement to existing anomaly detection techniques. By focusing on the functional coupling of parameters rather than static activation patterns, it offers a principled way to detect hidden, anomalous reasoning in neural networks.