PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

1. Problem Statement

Vision-Language Models (VLMs) are increasingly deployed in critical domains (medical imaging, autonomous driving, robotics), yet they remain highly vulnerable to adversarial image perturbations. These are small, human-imperceptible pixel-level changes that cause significant prediction shifts (e.g., misclassifying a "jeans" image as a "t-shirt").

Existing defense mechanisms suffer from four major limitations:

1. Gradient Dependency: Many methods (e.g., adversarial training) require white-box access to model gradients, which is impossible for closed-source VLMs accessed via APIs.
2. Poor Generalization: Training-time defenses often overfit to specific attack types (e.g., $\ell_\infty$-bounded attacks) and fail against unseen perturbations.
3. Restrictive Scope: Some approaches rely on specific architectural modifications or training objectives, limiting their applicability to general-purpose VLMs.
4. Clean Accuracy Trade-off: Improving robustness often degrades performance on clean, unperturbed inputs.

2. Methodology: The PDA Framework

The authors propose Paraphrase-Decomposition-Aggregation (PDA), a training-free, black-box defense framework that operates entirely at test time. It leverages text augmentation to stabilize VLM predictions without modifying the underlying model weights. The framework is inspired by Randomized Smoothing but applies it to the semantic text space rather than the image space.

The pipeline consists of three stages:

Step I: Paraphrase (Semantic Neighborhood Sampling)

• Goal: Explore the "neighborhood" of the original user query in the semantic text space.
• Mechanism: A Paraphrase Agent (typically an LLM) generates $N$ semantically equivalent but literally distinct prompts ($t_1, ..., t_N$) from the original query $t$.
• Rationale: Adversarial attacks often exploit the sensitivity of VLMs to specific visual-textual alignments. By varying the linguistic expression while preserving meaning, the system creates multiple "views" of the same task. If the image is perturbed, the model's response to one specific phrasing might be biased, but the consensus across diverse phrasings is likely to remain stable.

Step II: Decomposition (Atomic Question Generation)

• Goal: Break down complex queries into verifiable, atomic evidence.
• Mechanism: A Decomposition Agent (LLM) converts each paraphrased prompt into a sequence of $K$ simpler, atomic questions ($s_{i,1}, ..., s_{i,K}$).
• Rationale: Complex reasoning is prone to hallucination under attack. Decomposing a query (e.g., "What is the main object?") into atomic facts (e.g., "Is there a shirt?", "Is it on the upper body?", "What is the color?") forces the VLM to provide granular, image-aligned evidence. This makes the reasoning process more interpretable and robust.

Step III: Aggregation (Consensus Voting)

• Goal: Synthesize answers to suppress adversarially biased views.
• Mechanism:
  • The VLM answers all atomic questions for all paraphrases.
  • Structured Tasks (VQA/Classification): A lightweight decision head aggregates sub-answers via majority voting to produce a label for each paraphrase, then aggregates across paraphrases for the final result.
  • Open-Form Tasks (Captioning): An LLM aggregator summarizes the evidence set into a final caption, correcting inconsistencies.
• Rationale: Adversarial perturbations typically cause sporadic failures. By requiring agreement across multiple textual views and granular evidence, PDA filters out outliers caused by the attack.

Efficiency Variants

To balance robustness and inference cost, the authors introduce four variants:

1. PDA-Full: Full pipeline (Paraphrase $\to$ Decompose $\to$ Aggregate). Highest robustness, highest cost.
2. PDA-RJV (Reason-Judge-Vote): Skips decomposition; VLM answers paraphrases with rationales; LLM judges and votes.
3. PDA-RDA (Reason-Direct-Aggregate): Skips decomposition; LLM aggregates all paraphrase-rationale tuples in a single pass.
4. PDA-PV (Paraphrase-Vote): Skips decomposition and LLM aggregation; uses logically equivalent paraphrases and simple majority voting. Lowest cost.

3. Key Contributions

1. Novel Defense Paradigm: PDA is the first training-free, black-box framework specifically designed to defend VLMs against adversarial image attacks by manipulating the text input.
2. Generalizability: It is model-agnostic and task-agnostic, working across different VLM architectures (LLaVA, DeepSeek-VL, InternVL), parameter scales, and tasks (VQA, Classification, Captioning).
3. Robustness-Efficiency Trade-offs: The introduction of budgeted variants (RJV, RDA, PV) allows users to select a configuration that fits their latency constraints while retaining significant robustness gains.
4. Comprehensive Evaluation: The framework is validated against both white-box (PGD, EOT-Adaptive) and black-box (Transfer, Query-based) attacks, demonstrating consistent improvements.

4. Experimental Results

The authors evaluated PDA on VQA-v2, ImageNet-D, and MS COCO using models like LLaVA-1.5 (7B/13B), DeepSeek-VL, and InternVL3.

• White-Box Robustness: Under standard PGD attacks ($\epsilon=2/255$), PDA significantly improved adversarial accuracy.
  • Example (LLaVA-1.5-7B on VQA-v2): Adversarial accuracy increased from 44.5% (No Defense) to 75.4% (PDA), while clean accuracy remained nearly unchanged (86.2% $\to$ 87.1%).
  • Example (ImageNet-D): Clean accuracy actually improved (62.5% $\to$ 75.8%) because PDA reduced spurious predictions on clean inputs by cross-checking evidence.
• Adaptive Attacks: Even against Expectation over Transformations (EOT) attacks that explicitly optimize against PDA's stochastic components, the defense held up, maintaining a clear margin over the baseline.
• Black-Box & Closed-Source: PDA successfully defended closed-source APIs (GPT-5, Claude, Gemini) against transfer-based attacks, proving its utility in real-world deployment where gradients are unavailable.
• Comparison: PDA outperformed existing baselines including:
  • Text-side smoothing: (SmoothLLM, SemanticSmooth) which showed modest gains.
  • Image-side defenses: (JPEG, Random Augmentation) which were less effective than PDA.
  • Reasoning baselines: (Chain-of-Thought, Self-Consistency) which failed to correct perception-level errors caused by image perturbations.

5. Significance and Impact

• Practical Deployment: PDA offers a "plug-and-play" solution for securing VLMs in production environments without the need for retraining or access to model internals.
• Theoretical Insight: The work demonstrates that textual redundancy can effectively counteract visual adversarial perturbations. It suggests that the decision boundary of VLMs can be stabilized by sampling the semantic text space, similar to how randomized smoothing stabilizes image classifiers.
• Safety: By providing a robust, training-free defense, PDA addresses a critical security gap in the rapid deployment of multimodal AI systems, particularly in safety-critical fields like healthcare and autonomous driving.

In conclusion, PDA establishes a new standard for VLM robustness by shifting the defense strategy from modifying the visual encoder to leveraging the semantic flexibility of the language component, achieving strong robustness with minimal computational overhead.