💻 computer science

ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Models

ReACT-CLIP is a training-free, test-time defense for CLIP-style models that dynamically adapts correction strength and determines the necessity of intervention for each input by leveraging relative cross-noise feature drift and prediction instability, thereby significantly improving adversarial robustness across diverse attacks while preserving clean accuracy.

Original authors: Hashmat Shadab Malik, Toluwani Aremu, Samuele Poppi, Muzammal Naseer, Salman Khan

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hashmat Shadab Malik, Toluwani Aremu, Samuele Poppi, Muzammal Naseer, Salman Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just see pictures; they understand them by talking about them. This is the magic of Vision-Language Models, like the famous "CLIP." Think of CLIP as a super-smart librarian who has read every book and seen every photo in the universe. If you show it a picture of a golden retriever, it doesn't just recognize the dog; it instantly matches the image to the words "golden retriever" in its vast mental library. This ability to connect images and text without needing to be retrained for every new job has made these models the backbone of modern AI, powering everything from search engines to self-driving cars.

However, there's a catch. These super-librarians are surprisingly fragile. A sneaky attacker can add a tiny, almost invisible layer of "noise" to a picture—like a few pixels of static on a TV screen—that completely confuses the librarian. Suddenly, the librarian looks at a picture of a dog and confidently says, "That's a toaster!" This is called an "adversarial attack." It's a huge problem because if the AI can be tricked so easily, we can't trust it to make important decisions. Scientists have been trying to build shields to protect these models, but most of their current solutions are like wearing a heavy, one-size-fits-all suit of armor. It's great against a peashooter, but if the attacker uses a cannon, the armor is too weak. If the attacker only uses a peashooter, the armor is so heavy it slows the librarian down, making them miss easy tasks.

This is where a new method called ReACT-CLIP comes in, offering a clever, lightweight solution. Instead of wearing a fixed suit of armor, ReACT-CLIP acts like a highly responsive bodyguard who checks the threat level before deciding how to react. The researchers found that existing defenses fail because they use a "fixed correction strength." They try to fix every problem with the same amount of force, which leads to two bad outcomes: they either don't fix strong attacks enough, or they over-correct weak ones and ruin the answer for clean pictures.

ReACT-CLIP changes the game by being "response-aware." It doesn't guess the strength of the attack; it asks the model itself. Here's how it works: When a picture arrives, the system gives it two gentle "prods" with different amounts of digital noise. It watches how much the model's understanding of the picture "drifts" or wobbles under these prods.

If the picture is clean (a real dog), it stays steady even when prodded.
If the picture is a weak attack, it wobbles a little more.
If it's a strong attack, it wobbles wildly.

By measuring the difference in how much the picture wobbles between a light prod and a heavy prod, ReACT-CLIP gets a precise, sample-by-sample reading of how much help the picture needs. It then builds a custom "anchor" (a stable version of the picture) and pulls the confused image back toward the truth with just the right amount of force. If the picture is already doing fine, the system leaves it alone. If it's under heavy fire, the system pulls hard.

The paper shows that this approach is incredibly effective. Across 12 different datasets and under various types of attacks (from tiny nudges to massive distortions), ReACT-CLIP keeps the AI's accuracy high, often maintaining around 60% success even when attacks are very strong. In contrast, other defenses that use a fixed strategy see their accuracy crash as the attacks get stronger. The researchers also discovered that simply checking if the model is "confused" isn't enough; they had to combine the "wobble" measurement with a check on how stable the model's predictions are under slight changes to the image. This combination allows the system to avoid fixing pictures that don't need fixing, preserving the model's natural intelligence.

In short, ReACT-CLIP proves that you don't need to train a new model or know exactly how an attacker is planning to strike to defend against them. By simply listening to how the model reacts to a little bit of noise, the system can automatically adjust its defense, acting like a smart, adaptive shield that is strong enough to stop a cannon but gentle enough not to hurt a peashooter. This makes AI safer and more reliable without sacrificing its speed or accuracy.

Technical Summary: ReACT-CLIP

Problem Statement

Vision–language models (VLMs) like CLIP rely on frozen visual encoders for diverse downstream tasks, creating a concentrated point of failure against adversarial attacks. While adversarial training and prompt tuning offer robustness, they often require retraining, risk overfitting to specific attack distributions, and trade clean accuracy for robustness. Consequently, training-free test-time defenses have emerged as a practical alternative.

However, existing test-time defenses suffer from a fundamental limitation: they employ fixed-intervention strategies. Whether through fixed prompt updates, fixed counterattack budgets, or fixed feature-correction scales (e.g., a global Gaussian noise scale for anchor construction), these methods cannot adapt to the varying severity of attacks at inference time. The attack budget ( $\epsilon$ ) is unknown during deployment, and the required correction strength varies significantly across samples, even within the same nominal budget. As demonstrated in the paper, fixed configurations that work for weak attacks often "undershoot" strong attacks, while those tuned for strong attacks "overshoot" and degrade performance on clean or weakly perturbed inputs. This mismatch causes existing defenses to degrade sharply as attack strength increases.

Methodology: ReACT-CLIP

ReACT-CLIP (Response-Aware Correction at Test-Time for CLIP) addresses this by reframing test-time defense as a sample-specific decision process. Instead of applying a globally fixed correction, ReACT-CLIP determines two distinct properties for each input:

Whether defensive intervention is necessary.
How strongly the input should be corrected.

The method operates without modifying the pretrained model or requiring prompt training. It relies on two core mechanisms:

1. Response-Conditioned Anchor Construction (Determining Correction Strength)

To estimate the required correction strength, ReACT-CLIP probes the input with two levels of Gaussian noise ( $\sigma_{low}$ and $\sigma_{high}$ ) and measures the resulting drift in the visual feature space.

Observation: While absolute drift at a single noise level is ambiguous (adversarial features can appear deceptively stable under weak noise), the relative cross-noise drift ( $r(x)$ ) provides a clear, graded signal. It is defined as the relative increase in drift from low to high noise:
$r(x) = \frac{d_{high}(x) - d_{low}(x)}{d_{low}(x)}$
Mechanism: This relative drift serves as a sample-specific proxy for correction demand. ReACT-CLIP maps $r(x)$ to a sample-specific correction scale $\sigma_{corr}(x)$ using a linear mapping calibrated once on a single dataset (Caltech256).
Application: This scale is used to construct a noise-averaged feature anchor ( $f_{anc}$ ) from Gaussian-noised views. The original visual feature is then extrapolated toward and beyond this anchor. A small $r(x)$ yields a small noise scale and mild correction; a large $r(x)$ yields a larger scale and greater corrective reach.

2. Selective Defensive Intervention (Determining Necessity)

Applying correction unconditionally can harm clean inputs that are already correctly classified. To decide whether to intervene, ReACT-CLIP combines the relative drift with a prediction-instability score.

Observation: Clean inputs retain stable class-probability distributions under weak spatial augmentations, whereas adversarial inputs exhibit greater variation.
Mechanism: A prediction-instability score $J(x)$ is computed using Jensen–Shannon divergence between the original prediction and the prediction under a weak stochastic augmentation $T(x)$ .
Decision: The final defensive intervention score is $s_{int}(x) = r(x) + J(x)$ . If $s_{int}(x)$ exceeds a threshold $\tau$ , the correction is applied; otherwise, the original feature is retained. This allows the system to distinguish clean inputs from weakly attacked inputs, a regime where relative drift alone is often ambiguous.

Key Contributions

Response-Conditioned Anchor Construction: The paper identifies that globally fixed correction strengths are suboptimal. It introduces a method to convert relative cross-noise drift into a sample-specific correction scale, allowing the defensive reach to adapt to both the attack budget and individual sample characteristics without retraining.
Selective Defensive Intervention: The authors propose a fused intervention score combining relative drift and prediction instability. This improves the detection of adversarial inputs in the weak-attack regime while preserving clean accuracy, addressing the ambiguity that plagues drift-only detectors.
Extensive Robustness Evaluation: The method is evaluated across 12 downstream datasets, ImageNet, and its distribution-shifted variants under PGD-10, PGD-100, C&W-10, and AutoAttack.

Results

Robustness Gains: Across 12 downstream datasets under PGD-10 attacks, ReACT-CLIP maintains approximately 60% robust accuracy across attack budgets from $\epsilon=1/255$ to $\epsilon=16/255$ .
Comparison to Baselines: It outperforms the strongest existing baseline (Defend-CLIP) by up to 46.50 percentage points at $\epsilon=16/255$ . While existing defenses degrade sharply as attacks strengthen, ReACT-CLIP maintains stable performance.
Clean Accuracy: The method preserves clean accuracy, remaining within 0.76% of the strongest defense on clean inputs.
Generalization: The linear mapping for correction strength, calibrated on a single dataset, generalizes effectively to ImageNet and its out-of-distribution variants (ImageNet-A, V2, R, Sketch) and extrapolates to attack budgets ( $\epsilon=24, 32$ ) beyond the calibration range, where other defenses collapse.
Ablation: The results confirm that response-conditioned anchor construction drives robustness at moderate-to-strong attack budgets, while the selective intervention score primarily protects clean accuracy and improves performance in the weak-attack regime.

Significance and Claims

The paper claims that ReACT-CLIP establishes a new paradigm for training-free test-time defense by moving from global, fixed interventions to per-input, response-conditioned decisions. By leveraging the relative dynamics of noise-induced feature drift and prediction stability, the method adapts to unknown attack budgets and varying sample severities without requiring knowledge of the attack parameters at inference.

The authors emphasize that this approach allows a single, fixed configuration to remain effective across a wide spectrum of perturbation budgets, from very weak to very strong attacks, while largely preserving the zero-shot capabilities of the pretrained CLIP model. The work highlights that the "mismatch" between fixed defense configurations and variable attack strengths is a primary cause of failure in current test-time defenses, and that response-aware conditioning is a necessary step toward robust, deployable VLMs.