💻 computer science

Unifying Adversarially Robust Model Experts in Vision-Language Models

This paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies multiple specialized robust model experts through embedding-space harmonization to create a single vision-language model with superior, complementary adversarial robustness across diverse evaluation settings.

Original authors: Nguyen Duc Thai, Junhao Dong, Sua Qi Rong, Hua Yu, Yew-Soon Ong

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nguyen Duc Thai, Junhao Dong, Sua Qi Rong, Hua Yu, Yew-Soon Ong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can look at a picture and instantly tell you what it is, or even write a poem about it. This robot, known as a Vision-Language Model (VLM), is amazing because it understands both pictures and words at the same time. But like any smart kid, this robot has a weakness: it can be easily tricked. If someone adds a tiny, invisible speck of "noise" to a picture—like a few pixels shifted just a hair's breadth—the robot might suddenly think a panda is a toaster or a cat is a car. This is called an "adversarial attack," and it's a big problem because we want these robots to be safe and reliable in the real world.

To fix this, scientists have been teaching the robot to practice with these tricky pictures, a process called "adversarial training." Think of it like a martial arts student sparring with a partner to learn how to block punches. The paper you're about to read explores a fascinating discovery: there isn't just one way to teach the robot to block. Some teachers focus on memorizing the exact moves (learning to recognize specific words), while others focus on keeping their balance no matter what (keeping the picture's features steady). The problem is, a student who is great at memorizing moves might fall over if the fight changes, and a student who is great at balance might forget the specific names of the moves. This paper asks a simple but powerful question: What if we could get the best of both worlds?

The Tale of Two Experts and the Magic Glue

In the world of AI safety, researchers have been training "experts" to make Vision-Language Models tougher against attacks. But here's the catch: these experts are a bit like specialized athletes. One type of expert, let's call them the "Word-Whizzes," gets really good at matching pictures to specific text descriptions. If you show them a picture of a panda and ask, "Is this a panda?", they are rock-solid. But if you show them a picture of a new animal they've never seen before, they might stumble because they rely so heavily on the text they memorized.

The other type, the "Steady-Hands," focuses on keeping the picture itself looking the same, even when someone tries to mess it up. They are great at handling new, unseen animals because they don't care as much about the specific words; they just know what a "cat-like shape" looks like. However, they aren't as sharp at using the text clues to get the right answer on familiar animals.

For a long time, scientists tried to solve this by training one expert, then the other, and then trying to mash them together at the end. But it was like trying to glue two different types of clay together while they were still wet; they kept pulling apart, or the final result was a messy blob that wasn't good at anything.

Enter CARE: The Collaborative Coach

The authors of this paper, Nguyen Duc Thai and his team, decided to try something different. Instead of training one expert and then the other, they built a framework called CARE (Collaborative Adversarial Robustness fine-tuning using Embedding alignment). Think of CARE as a brilliant coach who puts the "Word-Whizz" and the "Steady-Hand" in the same gym and makes them train together, side-by-side.

Here is how the magic happens:

The Shared Warm-up: Before the experts start their specific drills, they share a "universal perturbation." Imagine they both look at the same slightly distorted picture to get a feel for the trouble. This helps them start on the same page.
The Specialized Drills: The "Word-Whizz" practices matching the distorted picture to the correct text. The "Steady-Hand" practices making sure the picture doesn't change its shape too much.
The Secret Handshake (Harmonization): This is the most important part. While they are training, the coach forces them to talk to each other. The "Word-Whizz" tells the "Steady-Hand," "Hey, look at how I'm matching the text!" and the "Steady-Hand" says, "Hey, look at how I'm keeping the picture stable!" They exchange this knowledge in real-time. This stops them from drifting too far apart and ensures they both learn from each other's strengths.
The Final Merge: At the end of the day, the coach blends their brains together into one super-expert. This new model isn't just a mix; it's a unified brain that knows how to match words and keep pictures stable.

What They Found

The team tested this new method on a famous dataset called ImageNet and many other tricky picture sets. The results were pretty cool.

Beating the Specialists: The new CARE model was better than the "Word-Whizz" alone and better than the "Steady-Hand" alone. In fact, when the attacks were strong (with a perturbation size of $\epsilon = 4/255$ ), CARE outperformed the "Word-Whizz" (TeCoA) by about 2% and the "Steady-Hand" (FARE) by nearly 8%.
Handling the Unknown: The CARE model didn't just get better at known animals; it also got better at guessing new ones. The "Steady-Hand" usually wins at new stuff, and the "Word-Whizz" usually wins at known stuff. CARE managed to be the best at both.
Real-World Tasks: They also tested the model on tasks like writing captions for pictures and answering questions about them. Even when the pictures were attacked, CARE kept giving the right answers more often than the other methods. For example, in a test where they had to identify a "Hot Dog" in a picture, CARE correctly said "Hot Dog" even under attack, while the others sometimes got confused.

The Cost of Being Great

Of course, there is a price to pay for this teamwork. Training two experts at once takes more computer power. The paper notes that training CARE takes about 1.5 times longer than training just one expert (83 hours vs. 48-50 hours) and uses more memory (52.6 GB vs. 25-29 GB). However, the authors suggest that with some clever engineering, like deleting temporary data as soon as it's used, they can keep the memory usage manageable.

The Bottom Line

The paper suggests that the old way of thinking—picking one strategy and sticking with it—might be too limiting. By letting different types of AI experts collaborate and learn from each other's strengths, we can build models that are much harder to trick. The authors don't claim this is the final solution to all AI safety problems, but they do show that this "collaborative" approach is a very promising direction. It's a reminder that sometimes, to be truly robust, you don't just need to be strong; you need to be able to work with others.

Technical Summary: Unifying Adversarially Robust Model Experts in Vision-Language Models

Problem Statement
Vision-Language Models (VLMs), such as CLIP, are highly susceptible to adversarial attacks, where imperceptible perturbations can drastically alter predictions. While adversarial fine-tuning is a primary defense strategy, existing methods typically produce specialized "robust model experts" with distinct and often conflicting characteristics. Specifically, the paper identifies two dominant fine-tuning philosophies:

Image-Text Alignment Experts (e.g., TeCoA): These rely on the shared embedding space to guide training, maximizing alignment between image and text prompts. They perform well on learned class names but struggle to maintain robustness on unseen classes or new text prompts.
Image-Invariance Experts (e.g., FARE): These focus on preserving visual representations by minimizing the distance between clean and adversarial image embeddings. They generalize better to unseen datasets but often underperform on known classes because they do not explicitly leverage the shared image-text representation.

The core challenge is that these experts diverge during training due to their different optimization objectives, making it unstable to simply merge them afterward. Previous attempts to combine models often require periodic parameter overwriting to maintain stability, which limits the exploration and specialization of individual experts.

Methodology: CARE Framework
The authors propose CARE (Collaborative Adversarial Robustness fine-tuning using Embedding alignment), a framework that trains multiple expert models simultaneously while facilitating knowledge exchange to prevent divergence. The framework integrates two specific experts: an Alignment expert (based on TeCoA) and an Invariance expert (based on FARE).

The CARE process consists of four key steps:

Universal Perturbation Initialization (GAIP): Instead of generating independent random noise for each expert, CARE uses a Global Adversarial perturbation Initialization Process (GAIP). This creates a shared universal perturbation by aligning the perturbed logits with the Alignment expert and the image embeddings with the Invariance expert using negative cosine similarity. This ensures a coherent starting point for adversarial generation.
Independent Adversarial Generation: Each expert generates its specific adversarial samples using the initialized perturbation. The Alignment expert uses Cross-Entropy (CE) loss, while the Invariance expert uses $L_2$ -distance loss.
Expert Harmonization: This is the core innovation. To prevent the experts from drifting apart, an Embedding-space harmonization mechanism is applied.
- The Invariance expert aligns its clean logits (which it normally does not optimize) with those of the Alignment expert.
- The Alignment expert aligns its clean image embeddings with those of the Invariance expert.
- Theoretical analysis (Theorem 1) suggests that using cosine similarity for these auxiliary losses acts as a momentum term, pulling the experts toward each other and stabilizing training without sacrificing their individual specializations.
Knowledge Generalization: Finally, the parameters of the unified model ( $\theta_{CARE}$ ) are updated using an Exponential Moving Average (EMA) of the two experts, weighted by a factor $\gamma$ .

Key Contributions

Unified Framework: The proposal of CARE, which maintains multiple adversarial robust experts and unifies their learned knowledge into a single model, addressing the instability of merging specialized models.
Harmonization Mechanism: The introduction of an embedding-space harmonization loss that enables knowledge exchange between experts during training. This allows the model to learn complementary robustness properties (both image-text alignment and image invariance) without the need for disruptive parameter overwriting.
Empirical Validation: Extensive experiments demonstrating that CARE outperforms individually trained experts (TeCoA and FARE) across both in-distribution image classification and zero-shot downstream vision-language tasks.

Results
The authors evaluated CARE on ImageNet-1K and 13 additional zero-shot datasets, as well as downstream tasks like image captioning (COCO, Flickr30k) and visual question answering (TextVQA, VQAv2).

Image Classification: CARE consistently outperformed both TeCoA and FARE. Under strong adversarial attacks ( $\epsilon = 4/255$ ), CARE outperformed TeCoA by approximately 2% and FARE by nearly 8% on average. Notably, CARE inherited the stability of TeCoA on in-distribution data while maintaining the generalization capabilities of FARE on unseen datasets.
Downstream Tasks: In image captioning and VQA tasks, CARE achieved the best robust accuracy across most settings. While FARE showed strong performance on clean data (due to its invariance nature), CARE provided superior robustness against adversarial perturbations, successfully transferring the alignment knowledge to different text encoders used in downstream models (LLaVA).
Qualitative Analysis: t-SNE visualizations showed that CARE produced compact clusters with well-defined boundaries, combining the high spread of TeCoA and the fragmentation of FARE. Qualitative captioning and VQA results demonstrated that CARE maintained consistent and correct outputs under attack, whereas other models often generated hallucinations or incorrect answers.

Significance and Claims
The paper claims that collaborative learning across model experts is a promising direction for improving adversarial robustness in VLMs. The significance of CARE lies in its ability to overcome the trade-off between in-distribution accuracy and zero-shot generalization. By harmonizing experts through embedding alignment, CARE creates a unified model that inherits the complementary strengths of different adversarial training strategies. The authors note that while the method requires additional computational resources (training multiple models), engineering optimizations like shared perturbation initialization and memory management mitigate these costs. The work suggests that future research should explore collaborative methods that account for the asymmetry between image embeddings and class logits, as the current alignment process showed that optimizing both does not always yield a strictly better model if not carefully balanced.

The Tale of Two Experts and the Magic Glue

Enter CARE: The Collaborative Coach

What They Found

The Cost of Being Great

The Bottom Line

More like this