🤖 AI

Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model

This paper proposes a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals to significantly enhance the robustness and cross-task generalizability of Large Vision-Language Models against adversarial attacks without requiring architecture modifications or task-specific retraining.

Original authors: Sibo Wang, Jie Zhang, Shiguang Shan, Xilin Chen, Wen Gao

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Sibo Wang, Jie Zhang, Shiguang Shan, Xilin Chen, Wen Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just see pictures but actually "understand" them, chatting about what's inside like a friend looking at a photo album. This is the realm of Large Vision-Language Models (LVLMs), the super-smart AI brains behind tools that can describe a sunset, answer questions about a diagram, or tell a story from a single image. To make these giants work, scientists teach them to connect images with words, much like matching a puzzle piece to its perfect spot. However, there's a sneaky problem: these AI brains are surprisingly fragile. Just as a magician can be tricked by a slight distraction, these models can be fooled by "adversarial attacks"—tiny, invisible tweaks to an image that look like static to a human eye but make the AI completely lose its mind, seeing a dog as a toaster or a bus as a giraffe. This is a big deal because as we start trusting these models with important jobs, we need to make sure they don't get tricked by digital noise.

Enter a new team of researchers who decided to stop the AI from being so easily fooled. They realized that previous attempts to "toughen up" these models were like training a boxer to only fight one specific opponent; the models got good at that one fight but couldn't handle anything else. The researchers proposed a clever new training camp called Dual Adversarial Fine-Tuning (DAFT). Instead of just teaching the AI to ignore the noise, they taught it two lessons at once. First, they showed it the "clean" version of the image (the truth) to keep its memory sharp. Second, they used rich, descriptive stories (captions) about the image instead of simple labels like "cat" or "dog." Think of it as teaching a student not just to memorize the word "apple," but to understand the whole concept of an apple—its redness, its crunch, and how it fits in a fruit bowl—so that even if someone scribbles on the picture, the student still knows what it is.

The paper finds that this double-pronged approach works wonders. When they tested their new method against the old ones, the DAFT-trained models stayed calm and collected even when the images were attacked with invisible noise. Whether the task was guessing what a picture was (classification), writing a sentence about it (captioning), or answering a tricky question (visual question answering), the new method kept its cool. The researchers showed that by replacing the "eyes" of the AI with their newly trained version, the whole system became much harder to trick, all without needing to rebuild the entire brain from scratch. It's a bit like giving a superhero a new set of goggles that filter out the bad stuff while keeping the good stuff crystal clear.

The Story of the Paper

The Problem: The "One-Trick Pony" AI
Large Vision-Language Models (LVLMs) are amazing. They can look at a photo of a Ferris wheel and tell you it's a "Ferris wheel," or answer a question like "What ride is in the middle?" But these models have a secret weakness: they are easily tricked. If you add a tiny bit of invisible static to a picture (an "adversarial attack"), the AI might suddenly think the Ferris wheel is a "bus" or a "giraffe."

Scientists have tried to fix this before. Some tried to train the AI using simple labels (like "cat" or "bus") while showing it these tricky, noisy pictures. But the paper argues this is like training a dog to only sit when you say "sit" in a quiet room; if you say it in a noisy park, the dog gets confused. These methods worked okay for simple tasks but failed when the AI had to do harder things like writing a story or answering complex questions. Other methods tried to teach the AI without any labels at all, but they didn't get strong enough to handle the really tricky attacks.

The Solution: A Two-Part Training Camp
The authors of this paper, Sibo Wang and his team, came up with a new way to train the AI called DAFT (Dual Adversarial Fine-Tuning). They realized that to make the AI truly robust, it needed two kinds of guidance at the same time, like a student studying with two different teachers.

The "Visual" Teacher (The Anchor): This part of the training uses a frozen, original version of the AI's "eyes" (the vision encoder). It acts like a reference guide. When the AI looks at a noisy, tricky picture, this teacher says, "Hey, remember what the real picture looks like? Don't forget the original features." This stops the AI from getting confused and overfitting (memorizing the noise instead of the truth).
The "Semantic" Teacher (The Storyteller): This is the big innovation. Instead of using simple labels like "bus" or "cat," this teacher uses full sentences or captions that describe the image in detail. For example, instead of just saying "bus," it says, "A blue bus driving down a rainy street." The AI is then trained to match the noisy picture with this rich description. This helps the AI understand the meaning and the context of the image, not just the category.

The magic happens because these two teachers work together. The "Visual" teacher keeps the AI grounded in reality, while the "Semantic" teacher teaches it to understand the deep meaning of the image, even when the picture is messed up.

The Results: Stronger and Smarter
The team tested their new method on a bunch of different tasks. They took a popular AI model called LLaVA and swapped out its original "eyes" for their new, toughened-up version. Then, they attacked it with invisible noise to see what would happen.

Zero-Shot Classification: When asked to identify objects in pictures it had never seen before, the DAFT model was much better at ignoring the noise. On average, it improved the accuracy by about 12% compared to the old methods that used simple labels.
Captioning and Q&A: This is where the new method really shined. When asked to write a description of a noisy image or answer a question, the old models often gave nonsense answers (like calling a crane a "giraffe"). The DAFT model, however, kept getting it right. For example, in one test, while other models failed to identify a "Ferris wheel" correctly under attack, the DAFT model nailed it.
The Trade-off: The paper notes that sometimes, making a model so tough against attacks can make it slightly worse at seeing clean pictures. However, DAFT managed to keep its performance on clean pictures almost as good as the best existing methods, while being much stronger against attacks.

What the Paper Rules Out
The authors are very clear about what doesn't work. They argue that simply using category labels (like "cat" or "bus") as the only guide is not enough because it leads to overfitting. They also show that just using unsupervised methods (learning without any text help) isn't strong enough against the most advanced attacks. Their experiments suggest that you really need that combination of visual grounding and rich semantic understanding to get the best results.

How Sure Are They?
The authors ran extensive experiments across many different datasets (like Caltech101, COCO, and VQAv2) and under different levels of noise (specifically, perturbation budgets of $\epsilon = 2/255$ and $\epsilon = 4/255$ ). They compared their method against the current best methods (like TeCoA and FARE) and found that DAFT consistently outperformed them. They didn't just simulate this; they actually trained the models and tested them on real-world tasks. The results are presented as measured facts from their experiments, showing a clear improvement in robustness.

The Bottom Line
This paper suggests that to make AI vision models truly safe and reliable, we can't just teach them to ignore noise; we have to teach them to understand the story behind the image while keeping a firm grip on the visual facts. By using a "dual" approach that combines a visual anchor with rich, descriptive captions, the researchers have built a model that is much harder to trick, making it a significant step forward for the safety of these powerful AI tools.

Technical Summary: Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model

Problem Statement

Large Vision-Language Models (LVLMs), such as LLaVA and GPT-4V, have demonstrated remarkable capabilities in reasoning and understanding multimodal data. However, they remain inherently vulnerable to visual adversarial attacks, where imperceptible noise in visual inputs can force models to generate erroneous outputs, such as incorrect captions or answers to visual questions.

Existing defense strategies face two critical limitations when applied to LVLMs:

Overfitting and Poor Generalization in Category-Aligned Supervision: Methods like TeCoA rely on fixed text embeddings (e.g., ImageNet class labels) for supervision. While effective for zero-shot classification within predefined categories, this approach lacks the semantic richness required for complex tasks like image captioning and Visual Question Answering (VQA). Consequently, robustness gains often fail to transfer to unseen datasets or multimodal tasks.
Limited Gains from Unsupervised Fine-Tuning: Unsupervised methods like FARE improve generalization by aligning adversarial and clean features without labeled data. However, they primarily focus on minimizing feature discrepancies and often fail to defend against sophisticated attacks that specifically target the image-text alignment mechanisms of LVLMs.

Furthermore, conducting adversarial training from scratch for large models is computationally prohibitive. Therefore, the challenge lies in developing an adversarial fine-tuning strategy that enhances robustness across diverse downstream tasks without requiring full-model retraining or separate task-specific adaptations.

Methodology: Dual Adversarial Fine-tuning (DAFT)

The authors propose DAFT, a dual adversarial fine-tuning framework designed to jointly optimize visual and semantic supervision signals. The method focuses on fine-tuning the vision encoder (specifically CLIP's vision encoder) of an LVLM. Once fine-tuned, this robust encoder can replace the original encoder in models like LLaVA, inheriting adversarial resilience without modifying the language model or retraining the entire system.

The framework consists of two synergistic branches:

1. Visual Supervision Branch

This branch aims to preserve the generalization capabilities of the pre-trained model and mitigate overfitting to specific adversarial examples.

Mechanism: It utilizes a frozen original vision encoder ( $F_{ori}$ ) to extract features from clean images.
Objective: The target model ( $F_\theta$ ) is trained to minimize the $L_2$ distance between the features of adversarial examples ( $x_a$ ) and the features of clean images ( $x$ ) extracted by the frozen encoder.
Loss Function: $L_{vis} = ||F_{ori}(x) - F_\theta(x_a)||_2^2$ .
Effect: This ensures the target model retains the robust, generalizable features of the original foundational model.

2. Semantic Supervision Branch

This branch addresses the need for rich semantic alignment to defend against attacks targeting multimodal coherence.

Mechanism: Instead of using simple category labels, this branch employs descriptive captions as the semantic supervision signal.
Objective: It aligns adversarial examples with their ground-truth descriptive captions while pushing them away from unrelated captions within the same batch.
Loss Function: A triplet loss is used where the adversarial feature is the anchor, the ground-truth caption embedding is the positive sample ( $P_o$ ), and the "hardest" negative sample ( $N_e$ )—defined as the caption in the batch most similar to the adversarial feature—is used to maximize the margin.
Loss Function: $L_{sem} = \max(\frac{F_\theta(x_a) \cdot T_\phi(N_e)}{||F_\theta(x_a)|| ||T_\phi(N_e)||} - \frac{F_\theta(x_a) \cdot T_\phi(P_o)}{||F_\theta(x_a)|| ||T_\phi(P_o)||} + \alpha, 0)$ .
Effect: This leverages the rich contextual information in captions to maintain semantic coherence under attack, which is crucial for tasks like VQA and captioning.

Optimization

The final objective function combines both branches: $L_{DAFT} = L_{sem} + \gamma L_{vis}$ . The training process involves an alternating min-max optimization:

Attack Generation: Generate adversarial examples $x_a$ by maximizing $L_{DAFT}$ using Projected Gradient Descent (PGD).
Parameter Update: Update the vision encoder parameters $\theta$ by minimizing $L_{DAFT}$ on the generated adversarial examples.

Key Contributions

Dual Supervision Framework: The proposal of DAFT, which jointly optimizes visual and semantic supervision signals to enhance cross-task robustness in LVLMs while mitigating overfitting.
Caption-Guided Semantic Supervision: To the authors' knowledge, this is the first framework to integrate descriptive captions (rather than class labels) as semantic supervision for adversarial training. This approach significantly improves robustness in multimodal tasks by leveraging richer semantic information.
Plug-and-Play Robustness: The method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original LVLM, requiring no separate task-specific retraining or architecture modifications.

Experimental Results

The authors evaluated DAFT on zero-shot classification, image captioning, and VQA tasks across multiple datasets (e.g., Caltech101, COCO, VQAv2) using perturbation budgets of $\epsilon = 2/255$ and $\epsilon = 4/255$ .

Zero-Shot Classification: DAFT outperformed state-of-the-art methods (TeCoA, PMG-AFT, FARE). Under $\epsilon = 2/255$ , it improved average robust accuracy by approximately 12% over category-label supervised methods and 1.2% over the unsupervised FARE method, while maintaining comparable clean accuracy.
Image Captioning and VQA: DAFT demonstrated superior robustness in generative tasks. Compared to FARE (the previous SOTA for generalization), DAFT improved average CIDEr scores by 2.3 and average VQA accuracy by 1.1% under $\epsilon = 2/255$ . Under stronger attacks ( $\epsilon = 4/255$ ), the improvements were even more pronounced.
Ablation Studies:
- Loss Components: Using both visual and semantic supervision yielded the best results. Using only semantic supervision led to significant overfitting, while visual supervision alone failed to maximize robustness.
- Supervision Type: Replacing descriptive captions with category labels resulted in a 12.4% drop in zero-shot classification accuracy and a 15.2 point drop in CIDEr scores, validating the superiority of caption-based supervision.
- Negative Sampling: Selecting the "hardest" negative sample (most similar to the adversarial feature) was critical; using "easiest" negatives resulted in near-zero robustness.
- Hyperparameters: Optimal performance was achieved with a margin $\alpha=0.2$ and a balancing coefficient $\gamma=0.5$ .

Significance and Claims

The paper claims that DAFT addresses the critical gap in current LVLM defense research, where existing methods either overfit to specific classification categories or fail to protect complex multimodal alignment mechanisms. By unifying visual and semantic alignment under attack conditions, DAFT provides a scalable solution that enhances adversarial robustness across diverse downstream tasks (classification, captioning, VQA) without the computational cost of full-model retraining. The authors position this work as a step toward securing the deployment of LVLMs in security-sensitive applications by ensuring consistent performance even when visual inputs are perturbed.

The authors note limitations, including the focus on CLIP-based models, the white-box nature of the defense, and the reliance on the COCO dataset for training, suggesting future work could explore non-CLIP architectures, black-box defenses, and larger caption datasets.