Technical Summary: Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models

1. Problem Statement

Vision Language Models (VLMs), such as CLIP, achieve strong performance by aligning visual and textual modalities in a joint embedding space. However, this tight cross-modal coupling creates a vulnerability: small adversarial perturbations to the input image can disrupt the semantic alignment between the image and its corresponding text, leading to misclassification.

Existing defense mechanisms suffer from specific limitations:

• Adversarial Fine-Tuning (AFT): While effective, it requires retraining the visual encoder, which is computationally expensive and risks distorting the pre-trained feature manifold, often degrading generalization on clean data.
• Adversarial Prompt Tuning (APT): These methods optimize soft prompts while keeping the backbone frozen. However, they typically generate static prompts shared across all inputs or rely on unidirectional projections (e.g., text-to-vision only). They fail to adapt to instance-specific perturbations and lack a mechanism for mutual correction between modalities.
• Test-Time Defenses: Many focus on visual-side consistency or attack characteristics rather than restoring the underlying cross-modal alignment.

The core limitation identified is that most defenses treat modalities independently or adjust only one branch while treating the other as fixed context. Since adversarial attacks specifically target the decoupling of these branches, a defense requiring bidirectional recovery—where visual and textual features actively interact to re-establish alignment—is necessary.

2. Methodology: Closed-Loop Bidirectional Prompting (CLBP)

The authors propose Closed-Loop Bidirectional Prompting (CLBP), a test-time defense that keeps pre-trained encoders frozen and introduces a dynamic feedback loop between the visual and textual branches.

Core Components

1. Semantic Anchor (Semantic Bootstrapping):
   To prevent the iterative updates from drifting away from the pre-trained semantic manifold, the loop is initialized and constrained by a "Semantic Anchor." This is a fixed text embedding derived from a generic class template (e.g., "a photo of a {class}"). It acts as a stable prior.

2. Bidirectional Feedback Loop:
   The method alternates between two lightweight adapters:

   • Text-to-Vision (T2V) Denoising: The current text embeddings (initially the anchor) are processed by a T2V adapter to generate visual prompts. These prompts are prepended to the image patches. The visual encoder then processes the image conditioned on these prompts, effectively filtering adversarial noise in the visual features through cross-attention against the stable language signal.
   • Vision-to-Text (V2T) Refinement: The resulting refined visual features are mapped by a V2T adapter into a per-instance shift ($\delta$) applied to the text prompt. This updates the textual description to be specific to the current image instance rather than shared across the dataset.

   This process forms a closed loop: $W^{(k)} \xrightarrow{T2V} z^{(k)} \xrightarrow{V2T} W^{(k+1)}$.

3. Theoretical Stability:
   The authors model the loop as a fixed-point map on the unit sphere. They prove that under the anchor initialization, the composed operator is a contraction with a local Lipschitz constant $q \approx 10^{-4}$. This implies:

   • The loop converges to a unique fixed point.
   • A single bidirectional update ($N=1$) is sufficient to reach the fixed point, as subsequent iterations yield negligible improvement.
   • The deviation from the clean fixed point grows linearly with input perturbation but does not depend on iteration depth, ensuring stability.

4. Multi-View Aggregation:
   To handle sampling noise from individual augmentations, CLBP aggregates predictions across $V$ random augmented views of the input. It uses a similarity-weighted softmax to suppress outlier views that disagree with the consensus, enhancing robustness without requiring retraining.

5. Training Objective:
   The adapters are trained using a composite loss on adversarial examples:

   • Cross-Entropy ($L_{ce}$): Standard classification loss.
   • Margin Loss ($L_{mar}$): Enforces a feature-space margin on adversarial logits.
   • KL Divergence ($L_{kl}$): Distills the clean prediction into the adversarial branch to preserve clean accuracy.

3. Key Contributions

• Bidirectional Feedback Mechanism: The paper identifies the independent treatment of modalities as a key weakness in existing defenses and proposes CLBP, a method where visual and textual features mutually condition and correct each other at inference time without modifying the frozen encoders.
• Semantic Anchor & Convergence Analysis: The introduction of a Semantic Anchor to constrain the iteration is complemented by a theoretical analysis proving the loop's contractive nature. The authors demonstrate empirically and theoretically that a single update is sufficient for convergence.
• Comprehensive Evaluation: The method is validated across 11 datasets under three regimes: cross-dataset zero-shot, few-shot adaptation, and base-to-new generalization.

4. Experimental Results

The authors evaluate CLBP against Adversarial Fine-Tuning (AFT), Adversarial Prompt Tuning (APT), and Test-Time defenses (e.g., TTC, SCC, R-TPT) under white-box attacks (PGD, AutoAttack, Carlini-Wagner, EOT-PGD).

• Cross-Dataset Zero-Shot Robustness: On 10 datasets, CLBP achieves an average robust accuracy of 52.77%, significantly outperforming the strongest test-time defense (SCC at 49.71%) and training-based baselines (e.g., AdvMaPLe at 24.37%). Crucially, it maintains a clean accuracy of 59.99%, nearly identical to vanilla CLIP (59.60%), avoiding the accuracy-robustness trade-off seen in AFT.
• Few-Shot Adaptation: In a 16-shot setting across 11 datasets, CLBP achieves 65.08% average robust accuracy, roughly double the performance of the next best baseline (FAP at 34.61%). In the low-shot regime (1-shot), CLBP retains 41.13% robustness, whereas static prompt baselines drop to ~2.43%.
• Base-to-New Generalization: CLBP demonstrates superior transferability, retaining 54.92% robust accuracy on unseen "New" classes, compared to 21.86% for FAP. This indicates the method does not overfit to base classes.
• Attack Diversity: CLBP maintains state-of-the-art performance against AutoAttack ensembles, Carlini-Wagner margin attacks, and adaptive EOT-PGD attacks, confirming the robustness is not an artifact of gradient obfuscation.
• Efficiency: The method is efficient; robust accuracy saturates with as few as 8 augmented views, and a single loop iteration is sufficient.

5. Significance and Claims

The paper claims that CLBP represents a significant step forward in securing VLMs by addressing the root cause of their vulnerability: the decoupling of cross-modal alignment under attack.

• Parameter Efficiency: Unlike AFT, CLBP does not require retraining the massive pre-trained encoders, making it suitable for resource-constrained environments or deployed systems.
• Dynamic Adaptation: By generating instance-specific prompts through a bidirectional loop, CLBP overcomes the static nature of previous prompt tuning methods, allowing it to adapt to diverse adversarial patterns.
• Theoretical Grounding: The work provides a theoretical justification for the stability of the defense, showing that the closed-loop dynamics are contractive and converge rapidly.
• Preservation of Clean Performance: The method achieves state-of-the-art robustness without sacrificing clean data accuracy, a common failure mode in adversarial training.

The authors acknowledge limitations, noting that the analysis focuses on dual-tower architectures (like CLIP) and that the theoretical bounds, while empirically tight, rely on factorized approximations. They also note that the method inherits biases from the underlying frozen backbone. Nevertheless, CLBP is presented as a robust, efficient, and theoretically grounded solution for adversarial defense in vision-language models.