💻 computer science

NeuroTrace: Inference Provenance-Based Detection of Adversarial Examples

NeuroTrace introduces a framework and benchmark dataset that leverages Inference Provenance Graphs to capture cross-layer execution structures, demonstrating that analyzing inference provenance offers a robust and transferable method for detecting adversarial examples across diverse domains.

Original authors: Firas Ben Hmida, Philemon Hailemariam, Kashif Ali Khan, Birhanu Eshete

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Firas Ben Hmida, Philemon Hailemariam, Kashif Ali Khan, Birhanu Eshete

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a highly sophisticated, black-box robot chef (a Deep Neural Network) that can perfectly identify ingredients in a photo or detect if a computer file is a virus. You trust it to make the right calls. But what if a hacker sneaks in a tiny, invisible speck of dust on the photo, or a single weird byte in the file, tricking the robot into thinking a "cat" is a "dog" or that a "virus" is "safe"?

This is the problem of Adversarial Examples. The robot makes a mistake, but to us, the input looks normal. The robot's internal logic is opaque; we can't see why it got confused.

Enter NeuroTrace, a new tool developed by researchers at the University of Michigan-Dearborn. Think of NeuroTrace not as a new chef, but as a forensic investigator that watches the chef's entire cooking process to catch the trickster.

Here is how it works, broken down into simple concepts:

1. The Problem: Looking at the Wrong Clues

Most current security systems try to catch hackers by looking at the final result (the robot's guess) or just one step in the process (like checking if the robot's "eyes" are wide open).

The Flaw: It's like trying to catch a pickpocket by only looking at their hands at the very end of the street. By the time you see the theft, it's too late, and the pickpocket might have changed their hands.
The Reality: Hackers don't just change the final answer; they mess up the entire flow of information inside the robot's brain.

2. The Solution: The "Inference Provenance Graph" (IPG)

NeuroTrace introduces a concept called an Inference Provenance Graph (IPG). Let's use an analogy:

Imagine the robot chef is a massive, multi-story factory.

Normal Operation: When a customer orders a "Cat," the signal travels down a specific, well-worn path: Raw Material -> Cutting Station -> Painting Station -> Packaging. The workers (neurons) along this path are active, and the conveyor belts (connections) are humming.
Adversarial Attack: A hacker sneaks in a tiny change. Suddenly, the signal takes a weird detour. Maybe it goes to the "Dog" painting station by mistake, or it activates a "Fire Alarm" worker who should be sleeping. The final product might still look like a "Cat" box, but the journey the signal took was chaotic and wrong.

NeuroTrace builds a map of this journey. It doesn't just look at the final box; it records:

Which workers were active?
Which conveyor belts were used?
How strong was the signal between them?

This map is the IPG. It captures the "DNA" of how the robot thought about that specific input.

3. The Detective Work: Spotting the "Systemic Taint"

The researchers found that even if a hacker is very clever, they can't help but leave a systemic stain on the robot's internal map.

The Analogy: Imagine a normal person walking through a crowded room. They take a natural path, bumping into a few people, avoiding others. A person running away from a crime (the hacker) might bump into everyone, run through walls, or take a path that makes no sense.
The Discovery: NeuroTrace uses a special type of AI (a Graph Neural Network) to read these maps. It learns that "Normal Maps" look like smooth, organized highways, while "Hacked Maps" look like chaotic, tangled webs. Even if the hacker uses a different trick (a different attack), the chaos in the map looks similar.

4. Why This is a Big Deal

The paper tested NeuroTrace on two very different worlds:

Vision: Identifying images (like CIFAR-10).
Malware: Detecting computer viruses.

The Results:

It works everywhere: Whether the robot is looking at pictures or scanning code, the "chaotic map" signature of a hack is the same.
It's transferable: If you train the detective on one type of hacker (e.g., someone who changes pixels), it can still catch a different type of hacker (e.g., someone who uses black-box tricks) because they both create the same kind of internal chaos.
It's better than the old way: It outperformed previous methods that only looked at isolated parts of the robot's brain.

5. The Catch (Trade-offs)

There is a cost to this super-vision.

The Speed: Creating this detailed map of the robot's journey takes time. It's like filming the entire factory in 4K resolution instead of just taking a photo of the finished product.
The Storage: These maps take up space.
The Verdict: Right now, NeuroTrace is best for high-stakes situations where you have a few seconds to check a critical file (like a bank transfer or a medical diagnosis) or for forensics (investigating why a system failed after the fact). It might be too slow for checking millions of spam emails in real-time, but it's incredibly powerful for catching the smartest, most dangerous hackers.

Summary

NeuroTrace is a new security system that stops trying to guess what the robot is thinking and starts watching how it thinks. By mapping the entire journey of information inside a neural network, it can spot the subtle, chaotic fingerprints of a hacker, even when the final result looks perfectly normal. It turns the "black box" of AI into a transparent, auditable process.

1. Problem Statement

Deep Neural Networks (DNNs) are increasingly opaque, making it difficult to detect and diagnose malicious inputs like adversarial examples. Existing detection methods primarily rely on layer-local signals (e.g., logits, gradients, saliency maps, or activation summaries). These approaches treat model components in isolation, failing to capture the cross-layer information flow and the structural execution patterns that occur during inference. Consequently, they often struggle to generalize across different attack types or threat models (white-box vs. black-box). The authors posit that adversarial perturbations cause systemic disruptions in the network's information flow that are not visible through local activation analysis alone.

2. Methodology: The NeuroTrace Framework

The paper introduces NeuroTrace, a unified framework for extracting, representing, and analyzing Inference Provenance Graphs (IPGs).

A. Inference Provenance Graphs (IPGs)

IPGs are heterogeneous graphs that capture the structured execution of a model during a forward pass. Unlike static computation graphs, IPGs are input-dependent subgraphs constructed dynamically.

Nodes: Represent intermediate activations (e.g., individual neurons in fully connected layers or channels in convolutional layers). Node features include activation statistics (mean, $L_2$ norm, sparsity) and metadata.
Edges: Represent computational dependencies between layers.
- Parametric edges: Weighted by learned parameters (weights).
- Non-parametric edges: Unit-weighted, representing structural connectivity (e.g., pooling, normalization).
Construction: The framework uses a thresholding mechanism ( $\|h_v\| \geq \tau$ ) to filter out low-magnitude activations, creating a compact subgraph that highlights the active computational pathways induced by a specific input.

B. Extraction Pipeline

NeuroTrace provides a reproducible extraction engine built on PyTorch:

Instrumentation: Uses forward hooks at module boundaries to intercept activations and record dependencies.
Graph Generation: Constructs heterogeneous graphs using PyTorch Geometric HeteroData.
Standardization: Projects node features to a fixed dimension and aggregates kernel weights at the channel level to ensure tractability.

C. Detection Mechanism

The extracted IPGs are fed into a downstream Graph Neural Network (GNN) (specifically a 3-layer GraphSAGE) to classify inputs as benign or adversarial. The GNN learns to identify structural deviations in the information flow rather than just local activation anomalies.

3. Key Contributions

Inference Provenance as a Signal: The paper formalizes inference provenance as a graph-based representation of model execution, demonstrating that it captures cross-layer dependencies inaccessible to layer-local methods.
NeuroTrace Extraction Framework: A reproducible, architecture-agnostic pipeline that instruments model execution to construct heterogeneous IPGs with minimal adaptation.
NeuroTrace-IPG Dataset & Benchmark: The release of a multi-domain dataset covering:
- Domains: Image classification (CIFAR-10 with ResNet-20) and Malware detection (EMBER and Cuckoo-DNN).
- Attacks: Six attack families including white-box (FGSM, PGD, APGD) and black-box (SPSA, Square, SIA/SIT) attacks.
- Splits: Strict train/validation/test splits and cross-attack transfer splits.
Transferable Detection: Empirical evidence showing that IPG-based detectors generalize across unseen attack types and threat models (white-box to black-box and vice versa).

4. Experimental Results

The authors evaluated NeuroTrace under three protocols: Intra-attack, Multi-attack, and Cross-threat transfer.

Intra-Attack Performance:
- On CIFAR-10, NeuroTrace achieved >96% accuracy and >99% ROC-AUC across all six attack families.
- On Malware datasets, it achieved 100% accuracy on both Emb-att and Bit-Flip attacks, demonstrating domain versatility beyond vision.
Multi-Attack Training:
- A single detector trained on a mixture of all attack families achieved near-saturation performance (e.g., 99.75% - 100% accuracy) when tested on individual attacks, indicating the model learns a unified representation of adversarial structural deviations.
Cross-Threat Transfer (White-box $\leftrightarrow$ Black-box):
- Detectors trained on white-box attacks successfully detected black-box attacks (and vice versa) with >96% accuracy and near-perfect AUC. This confirms that adversarial perturbations induce shared structural deviations regardless of the generation mechanism.
Baseline Comparison:
- NeuroTrace significantly outperformed CIGA (Critical Inference Graphs), a state-of-the-art graph-based baseline, particularly on PGD and APGD attacks (e.g., 99.75% vs. 90.09% on PGD).

5. Runtime and Storage Analysis

Overhead: Extraction is linear with graph size.
- Vision Models (ResNet-20): ~15–25 ms per sample (batched) on a V100 GPU; serialized graph size ~5.3 MB.
- Malware Models: ~4–25 ms; serialized size ~0.98–6.15 MB.
Implication: While currently suitable for offline auditing, forensic analysis, and high-assurance settings, the overhead suggests a need for compression or selective capture for low-latency, real-time deployment on large architectures.

6. Significance and Conclusion

NeuroTrace establishes inference provenance as a robust, transferable signal for adversarial detection.

Theoretical Insight: It validates the hypothesis that adversarial examples cause systemic tainting of the inference process, altering the global information flow structure rather than just local activations.
Practical Impact: By releasing the extraction pipeline, dataset, and evaluation protocol, NeuroTrace enables systematic study of DNN behavior. It moves the field toward more transparent and auditable machine learning systems where the "why" and "how" of a prediction can be traced and verified.
Future Directions: The authors identify the need to evaluate against adaptive adversaries (who optimize specifically to evade the detector), extend the framework to larger architectures (ViTs, LLMs), and optimize graph construction for real-time latency.