Technical Summary: From Internal Diagnosis to External Auditing

1. Problem Statement

Deep Neural Networks (DNNs) remain fundamentally vulnerable to backdoor attacks, where adversaries poison training data to implant hidden triggers that hijack model behavior on specific inputs while preserving performance on benign data. The threat landscape has evolved from conspicuous patterns to stealthy dynamic, clean-label, and clean-image strategies, rendering traditional detection increasingly difficult.

In "Model-as-a-Service" scenarios, defenders often lack access to clean training data or the victim model's weights, necessitating data-free test-time defense. Current research is constrained by the internal diagnosis paradigm, which suffers from two fundamental vulnerabilities:

1. Model Repairing Methods: These inspect the victim model's internal properties (e.g., neuron activation traces, Channel Lipschitzness). They fail against advanced dynamic attacks that explicitly optimize triggers to suppress these internal anomalies.
2. Input Robustness Methods: These rely on input perturbations or consistency checks. They fail against natural backdoors (e.g., clean-image attacks) where the trigger is a robust semantic feature indistinguishable from benign features via perturbation.

The core challenge is developing a defense that is robust across diverse attack modalities without relying on the compromised model's internal states or fragile input assumptions.

2. Methodology: PRISM

The authors propose PRISM (Prototype Refinement & Inspection via Statistical Monitoring), a paradigm shift from internal diagnosis to Online External Semantic Auditing. This approach decouples defense from the compromised model by using a universal, pre-trained Vision-Language Model (VLM) as an independent, "clean" auditor.

PRISM operates as a test-time wrapper around the suspicious victim model using a three-pillar architecture:

A. Dual Stream Inference Framework

For every incoming test sample $x$, the system initiates two concurrent forward passes:

1. Victim Stream: Processes $x$ through the potentially compromised model $f_S$ to obtain predictions and logits.
2. Auditor Stream: Processes $x$ through a frozen, trusted Universal VLM (e.g., CLIP, Qwen-VL). The VLM acts as a "Universal Teacher," providing a semantic verification signal based on its vast, open-world knowledge, which is statistically independent of the victim's poisoned distribution.

B. Hybrid VLM Teacher

To bridge the domain gap between general VLMs and specialized downstream tasks, PRISM employs a hybrid anchoring mechanism:

• Static Text Anchors: Provide high-level semantic guidance via cosine similarity with predefined text labels.
• Dynamic Visual Prototypes: Augment the auditor with class centroids ($C$) learned online from the test stream.
• Fusion: The final VLM logits ($S_{VLM}$) are a weighted fusion of text-based and prototype-based similarities:
  $$S_{VLM} = \lambda \cdot \text{Sim}(z_{img}, C) + (1 - \lambda) \cdot \text{Sim}(z_{img}, T)$$
  This allows the auditor to adapt to the specific feature distribution of the test stream without requiring labeled training data.

C. Adaptive Router via Statistical Monitoring

The core decision logic determines whether to accept the victim's prediction or reject it in favor of the VLM. This is governed by a logit margin discrepancy metric ($\Delta$):
$$\Delta = \exp\left(S_{VLM}^{\hat{y}_S} - \max_{i \neq \hat{y}_S} S_{VLM}^i\right)$$
where $\hat{y}_S$ is the victim's predicted class. A small $\Delta$ indicates strong disagreement (potential attack), while a large $\Delta$ indicates alignment.

To handle the non-Gaussian, skewed nature of the $\Delta$ distribution, PRISM uses an Adaptive Threshold derived via the Cornish-Fisher expansion:
$$\tau_k = \mu_k + \left(\zeta + \frac{\gamma_k}{6}(\zeta^2 - 1)\right) \cdot \sigma_k$$
where $\mu_k, \sigma_k, \gamma_k$ are the running mean, standard deviation, and skewness of the margin for class $k$, and $\zeta$ is a base confidence coefficient.

D. Online Update Mechanism

PRISM employs a Cumulative Moving Average (CMA) to update statistical moments and visual prototypes. Crucially, updates occur only when a sample is certified as clean (i.e., $\Delta > \tau$). This selective update mechanism, combined with the statistical inertia of CMA, ensures robustness against distribution shifts and prevents poisoned samples from corrupting the defense state.

3. Key Contributions

1. New Paradigm (External Semantic Auditing): The paper identifies failure modes of internal diagnosis and proposes using universal VLMs as independent, external auditors, effectively decoupling defense from compromised victim models without accessing weights or training data.
2. Online Adaptive Framework (PRISM): A test-time framework that transforms generic VLMs into effective auditors via a Hybrid VLM Teacher (bridging the domain gap) and an Adaptive Router (avoiding static thresholds via statistical monitoring).
3. Superior Generalization and Robustness: PRISM achieves state-of-the-art performance across 17 datasets and 11 attack types, demonstrating unique robustness against advanced clean-image and adaptive attacks where traditional paradigms fail.

4. Experimental Results

Evaluation was conducted across 17 datasets (including CIFAR-10, CIFAR-100, ImageNet, GTSRB, and domain-specific datasets) and 11 state-of-the-art backdoor attacks (Classic, Dynamic, Clean-Label, and Clean-Image).

• Performance: PRISM successfully suppresses the Attack Success Rate (ASR) to < 1% on CIFAR-10 and consistently below 15% across all 11 attack types, while simultaneously improving or maintaining Clean Accuracy (CA).
• Comparison: It outperforms existing data-free defenses (e.g., CLP, ScaleUp, ZIP, Refine), which often fail completely against Clean-Image (FLIP, GCB) or Dynamic attacks. Naive VLM baselines achieve low ASR but suffer catastrophic CA drops (e.g., >70% on GTSRB), whereas PRISM bridges this gap.
• Efficiency: On an NVIDIA A100 GPU, PRISM (using CLIP) achieves an inference latency of 12.5 ms per image, ranking among the top defenses in efficiency.
• Robustness:
  • VLM Architectures: PRISM maintains low ASR (<10%) across both embedding models (CLIP, SigLIP) and generative models (Qwen, LLaVA).
  • Dataset Scale: It remains effective across dataset sizes varying by three orders of magnitude (200 to 10,000 images).
  • Adaptive Attacks: PRISM resists online adaptive strategies (Flooding, Periodic, Mixed) where the adversary attempts to manipulate the test stream, maintaining ASR < 20%.
  • Auditor Compromise: In a "Trust Chain" scenario where a specialized auditor is backdoored, a weak but clean generalist VLM can sanitize the stream, recovering security.

5. Significance and Claims

The paper claims that PRISM establishes that decoupling defense through external semantic auditing is not only feasible but essential for next-generation model security.

• Paradigm Shift: It moves away from the fragile assumption that backdoors leave conspicuous internal footprints or that triggers are sensitive to input perturbations. Instead, it leverages the statistical independence of foundation models to detect semantic discrepancies.
• Practicality: By operating in a data-free, test-time manner without modifying model weights, PRISM addresses the critical constraints of Model-as-a-Service deployments where defenders lack access to training data or model internals.
• Scalability: The framework is model-agnostic and can be extended to other tasks (e.g., image captioning) by adapting the logit-margin gating to sequence-level confidence, offering a scalable pathway for AI safety monitoring in safety-critical domains like healthcare and autonomous driving.

The authors conclude that while the paradigm relies on the statistical independence of the auditing VLM, it provides a robust safeguard against evolving threats, including stealthy clean-image and dynamic attacks, setting a new standard for model-agnostic defense.