🤖 AI

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

The paper proposes SS-TPT, a test-time prompt tuning method for vision-language models that enhances adversarial robustness and efficiency by evaluating augmented views through stability and suitability scores to guide adaptation and inference, thereby achieving superior robustness-throughput trade-offs compared to existing state-of-the-art approaches.

Original authors: Sunoh Kim, Daeho Um

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sunoh Kim, Daeho Um

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the AI model) who can identify objects in photos just by looking at them, even without being taught specific examples beforehand. This librarian is great at their job, but they are easily confused if someone sneaks a tiny, almost invisible smudge onto a photo (an "adversarial attack") or if the photo looks very different from the books in the library (a "distribution shift").

To help the librarian make better decisions, previous methods tried to show them many slightly different versions of the same photo (like showing a photo, a blurry version, a brightened version, and a rotated version). The idea was that if the librarian agrees on the answer across all these versions, they must be right.

The Problem:
Showing too many versions is slow. It's like asking a librarian to read 63 different copies of a book just to decide if it's a mystery novel. It takes forever, and the library gets backed up. Also, if one of those copies is a "bad" copy (distorted by the attacker), the librarian might get confused by it, even if they are looking at 62 good copies.

The Solution: SS-TPT
The authors of this paper created a new method called SS-TPT (Stability and Suitability-Guided Test-Time Prompt Tuning). Instead of just showing the librarian more copies, they teach the librarian how to judge the quality of each copy they see.

They use two simple "report cards" for every version of the photo:

Stability (The "Shake Test"):
Imagine you hand the librarian a photo and ask, "What is this?" Then, you gently shake the photo or slightly change the lighting (weak augmentations) and ask again.
- High Stability: The librarian says, "It's a cat," both times. They are confident and consistent.
- Low Stability: The librarian says, "It's a cat," then "It's a dog," then "It's a toaster." They are confused.
- The Lesson: If a view changes its mind easily with a tiny nudge, it's probably a bad view.
Suitability (The "Crowd Test"):
Imagine the librarian looks at all the photo versions together in a mental "feature space" (a map of how similar things look).
- High Suitability: The photo version is standing in a crowded group with its friends. Everyone looks similar.
- Low Suitability: The photo version is standing all alone in an empty field, far away from the others. It's an outlier.
- The Lesson: If a view is an oddball that doesn't fit with the others, it's likely corrupted or fake.

How It Works in Practice:
The SS-TPT system uses these two scores to act like a smart filter:

During Learning (Adaptation): When the librarian is trying to adjust their thinking to the new photo, the system tells them: "Ignore the views that are unstable or lonely. Only listen to the views that are stable and fit in with the crowd." This prevents the librarian from learning from the "bad" copies.
During the Final Guess (Inference): When it's time to give the final answer, the system doesn't just take a simple average of all guesses. Instead, it gives a louder voice to the trustworthy views (high scores) and a whisper to the untrustworthy ones (low scores).

The Results:
The paper claims this approach is a game-changer because:

It's Fast: You don't need to show the librarian 63 versions. You can get great results with just 15, and the system runs about twice as fast as previous methods.
It's Stronger: Even with fewer views, the librarian is much harder to trick by attackers. They achieve higher accuracy on tricky, attacked photos than any previous method.
It's Smart: It doesn't just count votes; it judges the quality of the voters.

In short, SS-TPT stops the AI from wasting time on bad copies of a photo and focuses only on the reliable ones, making it faster, smarter, and harder to fool.

Technical Summary: SS-TPT

Problem Statement
Vision-language models (VLMs), particularly CLIP, demonstrate strong zero-shot recognition capabilities but remain highly vulnerable to adversarial perturbations and distribution shifts. While recent test-time adaptation (TTA) defenses attempt to improve robustness by leveraging multiple augmented views, they introduce a critical inefficiency: a severe trade-off between robustness and throughput. Existing methods, such as R-TPT, often require a large number of augmented views (e.g., 63 views) to stabilize adaptation, resulting in inference speeds nearly four times slower than vanilla CLIP. Conversely, reducing the number of views to improve speed often degrades robustness significantly because these methods treat all views as equally trustworthy, allowing corrupted or outlier views to dominate the adaptation process. The core challenge is to achieve high adversarial robustness without sacrificing computational efficiency or requiring excessive visual diversity.

Methodology: SS-TPT
The authors propose Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), a framework that evaluates the quality of each augmented view to guide both model adaptation and inference. Instead of relying on the quantity of views, SS-TPT computes two complementary scores for every view $x_i$ :

Stability Score ( $S_{stab}$ ): Measures the invariance of a view's predictive distribution under weak stochastic augmentations (e.g., affine transformations, color jitter, blur, noise). It is defined as the inverse of the Jensen-Shannon divergence between the original view's prediction and its weakly augmented variant. High stability indicates that the view's prediction is robust to minor perturbations.
Suitability Score ( $S_{suit}$ ): Measures the density of a view within the feature space relative to other views. It is calculated as the inverse of the mean cosine-induced distance between a view's normalized feature vector and those of all other views. High suitability indicates that the view lies in a dense cluster of other views, suggesting it is not an outlier.

These scores are normalized and combined into a single Stability-Suitability (SS) score ( $Z_i$ ) via a hyperparameter $\alpha$ . This score is converted into SS weights ( $w_i$ ) using a softmax function.

The framework utilizes these weights in two distinct phases:

Adaptation (SS-Guided Consistency Loss): During test-time prompt tuning, the model minimizes a loss function combining point-wise entropy minimization with a consistency term. Unlike previous methods that either enforce consistency across all views (ignoring quality) or remove consistency entirely to avoid corrupted views, SS-TPT enforces alignment only with trustworthy references. Specifically, each view is aligned to a leave-one-out weighted average of other views, where the weights are determined by their SS scores. This prevents corrupted views from misleading the adaptation.
Inference (SS-Weighted Prediction): For the final prediction, the model aggregates the predictions of all views using the SS weights. This suppresses the influence of low-quality (corrupted or outlier) views while amplifying trustworthy ones.

Key Contributions

SS-TPT Framework: The introduction of a test-time defense that explicitly quantifies view quality through stability and suitability scores, enabling robust adaptation even with a limited number of augmented views.
Dual-Mechanism Design: The design of an SS-guided consistency loss that aligns predictions with trustworthy references during adaptation, and an SS-weighted prediction mechanism that suppresses outliers during inference.
Superior Robustness-Efficiency Trade-off: Demonstrating that focusing on view quality over quantity allows the model to outperform state-of-the-art methods in both robustness and throughput, achieving high robustness with significantly fewer views (e.g., 15 views vs. 63).

Experimental Results
Extensive experiments were conducted across 10 fine-grained datasets, ImageNet, and four distribution-shifted variants (ImageNet-A, V2, R, S) under various attack scenarios (PGD, CW, AutoAttack, White-box).

Robustness: SS-TPT achieved state-of-the-art adversarial robustness. On 10 fine-grained datasets, it reached an average robustness of 46.8%, surpassing the previous best (R-TPT) by 6.4 percentage points. On ImageNet, it achieved 48.6% robust accuracy, outperforming R-TPT by 5.4 points.
Clean Accuracy: The method preserved high clean accuracy (55.1% on fine-grained datasets), remaining closest to the original CLIP performance among all adaptation methods.
Efficiency: SS-TPT significantly reduced inference time. Using only 15 augmented views, it processed images in approximately 0.74 seconds, nearly twice as fast as R-TPT (which used 63 views and took 1.67 seconds), while achieving higher robustness (81.0% on Caltech101 vs. 79.8% for R-TPT-63v).
Low-View Regime: Even in the challenging single-view setting (1 augmented view), SS-TPT maintained an average robustness of 25.8%, outperforming R-TPT by 15.7 points, demonstrating its ability to extract meaningful information from limited visual evidence.
Ablation Studies: Analysis confirmed that both stability and suitability are necessary components, with the combination of SS-guided consistency and SS-weighted prediction yielding the best synergy. The method also proved robust to variations in prompt templates.

Significance and Claims
The paper claims that SS-TPT addresses the fundamental limitation of current TTA defenses: the reliance on large numbers of views to average out noise, which creates an impractical computational bottleneck. By introducing a mechanism to assess and weight view quality, SS-TPT decouples robustness from the sheer volume of augmented views. The authors assert that this approach offers a "practical and general" solution for deploying trustworthy VLMs in real-world scenarios where inputs may be noisy or manipulated, and where computational resources are constrained. The work highlights that view quality assessment is a critical, previously underutilized factor in test-time adaptation, establishing stability and suitability as effective indicators for distinguishing trustworthy views from corrupted ones.

Technical Summary: SS-TPT

More like this