Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models

This paper proposes MPCAttack, a novel framework that enhances the transferability of adversarial attacks against Multi-Modal Large Language Models by leveraging a Multi-Paradigm Collaborative Optimisation strategy to jointly aggregate and balance visual and textual semantic representations for more effective global perturbation.

The Gist

Imagine you have a super-smart robot friend (a Multi-Modal Large Language Model, or MLLM) that can look at a picture and tell you a story about it. It's like a detective that combines what it sees with what it knows about the world.

But, just like any smart person, this robot can be tricked. If you show it a picture with a tiny, almost invisible sticker on it (an adversarial perturbation), the robot might suddenly think a picture of a cat is actually a dog, or that a living room is a kitchen.

The problem is that most people trying to trick these robots use a "one-trick pony" approach. They try to confuse the robot using only one way of thinking (like only looking at colors, or only looking at shapes). But because the robot is so smart and uses many ways of thinking, a single trick often fails. The robot sees through it.

The New Idea: The "Swiss Army Knife" Attack

This paper introduces a new method called MPCAttack. Instead of using just one trick, the researchers built a "Swiss Army Knife" of tricks. They realized that to really fool a super-smart robot, you need to attack it from multiple angles at the same time.

Here is how they did it, using some fun analogies:

1. The Three Different "Detectives" (The Paradigms)

Imagine you are trying to trick a security guard. Most attackers hire just one detective to figure out the guard's routine. This paper says, "Let's hire three different types of detectives, each with a unique superpower":

• Detective A (The Matchmaker): This detective is great at matching pictures to words. If you show it a photo of a beach, it knows the word "sand" fits perfectly. It focuses on how well the image and text line up.
• Detective B (The Storyteller): This detective doesn't just match words; it understands the story. It knows that a picture of a beach implies "vacation," "sun," and "relaxation." It looks at the deep meaning and relationships between things.
• Detective C (The Pattern Spotter): This detective is a master of visual patterns. It doesn't care about words; it just sees shapes, textures, and lighting. It knows that a specific pattern of pixels usually means "sky."

2. The "Team Huddle" (Collaborative Optimization)

In the past, attackers would ask Detective A for a trick, then ask Detective B for a different trick, and just mash them together. It was like a team where everyone shouted their own ideas without listening to each other. The result was a messy, confused attack that the robot could easily spot.

MPCAttack changes the game by making the detectives hold a Team Huddle.

• They compare notes.
• They say, "Hey, Detective A, your trick is good for matching words, but Detective C's trick is better for hiding the shape. Let's combine them."
• They create a single, perfect trick that satisfies all three detectives at once.

This "huddle" ensures the trick isn't just good at fooling one type of thinking; it's good at fooling all types of thinking simultaneously.

3. The Result: A Master Illusionist

Because this new attack method (MPCAttack) uses all three perspectives together, the "sticker" it puts on the image is incredibly powerful.

• The Old Way: Like trying to trick a guard by wearing a fake mustache. The guard might look past it.
• The MPCAttack Way: Like a magician who changes the lighting, the music, and the costume all at once. The guard is so overwhelmed by the combined effect that they completely believe the illusion.

Why Does This Matter?

The researchers tested this on both open-source robots (like LLaVA) and famous closed-source robots (like GPT-4o and GPT-5).

• The Results: The new method was a huge success. It fooled the robots much more often than any previous method.
• The Lesson: It turns out that to break a complex system, you can't just use a hammer. You need a whole toolbox, and you need to use the tools together, not separately.

In a Nutshell

This paper teaches us that to understand (or break) the security of advanced AI, we need to stop thinking in just one way. By combining different "languages" of vision and understanding, we can create attacks that are much harder to defend against. This helps scientists find the weak spots in AI before bad actors do, making our future AI systems safer.

The takeaway: If you want to trick a genius, don't just use one argument. Use a team of experts who talk to each other to build the ultimate, unbreakable argument.

Technical Summary

1. Problem Statement

Multi-Modal Large Language Models (MLLMs) have revolutionized AI by integrating visual and textual reasoning. However, they remain highly vulnerable to adversarial attacks. Existing transferable adversarial attacks against MLLMs suffer from two critical limitations:

• Single-Paradigm Representation Constraint: Current methods typically rely on surrogate models trained under a single learning paradigm (e.g., only cross-modal alignment like CLIP). This confines the optimization to a limited feature space, causing perturbations to overfit the specific biases of that paradigm and resulting in poor generalization across diverse MLLM architectures.
• Independent Feature Optimization: Existing ensemble methods often treat features from different surrogate models as independent objectives, aggregating them via simple fusion. This ignores the semantic complementarity between different representation spaces, leading to redundant gradient directions, local optima, and weakened transferability.

2. Methodology: MPCAttack

The authors propose MPCAttack, a novel framework designed to enhance the transferability of adversarial examples by leveraging Multi-Paradigm Collaborative Optimization (MPCO).

Core Components

1. Multi-Paradigm Feature Aggregation:
   Instead of relying on a single model, MPCAttack integrates features from three distinct large-scale learning paradigms:

   • Cross-Modal Alignment: (e.g., CLIP) Focuses on matching image-text pairs in a shared semantic space.
   • Multi-Modal Understanding: (e.g., InternVL) Integrates visual and textual representations for deep reasoning and generation.
   • Visual Self-Supervised Learning: (e.g., DINOv2) Learns robust visual features from unlabeled data, emphasizing low-level cues and structural encoding.

2. Feature Fusion Strategy:

   • The framework extracts image features ($z^I, z^m, z^v$) from the three paradigms.
   • For the cross-modal paradigm, it also generates text descriptions of the source and target images using a text generator, encodes them, and fuses them with image features ($z^c$) using a weighting factor $\lambda$.
   • All features are $\ell_2$-normalized and concatenated to form a comprehensive Multi-Paradigm Feature Representation ($z_{adv}, z_s, z_t$).

3. Multi-Paradigm Collaborative Optimization (MPCO):

   • Contrastive Matching: The core innovation is a contrastive loss function that operates on the aggregated features. It minimizes the distance between the adversarial feature ($z_{adv}$) and the target feature ($z_t$) while maximizing the distance from the source feature ($z_s$).
   • Global Optimization: By optimizing against the joint representation of all paradigms, the method adaptively balances the importance of different features. This guides the perturbation toward a global semantic optimum, alleviating the bias inherent in single-paradigm approaches and preventing the optimization from getting stuck in local minima.

4. Attack Formulation:
   The objective is to find a perturbation $\delta$ such that the adversarial image $x_{adv} = x_s + \delta$ fools the black-box target MLLM. The optimization minimizes the contrastive loss subject to an $\ell_\infty$ norm constraint ($\|\delta\|_\infty \le \epsilon$).

3. Key Contributions

• Novel Framework: Introduction of MPCAttack, the first framework to jointly optimize adversarial perturbations using features from multiple, distinct learning paradigms (alignment, understanding, and self-supervised).
• Collaborative Strategy: Development of the MPCO strategy, which uses contrastive matching on aggregated features to adaptively balance paradigm representations and guide global perturbation optimization.
• Comprehensive Evaluation: Extensive experiments demonstrating that MPCAttack outperforms state-of-the-art (SOTA) methods on both open-source (e.g., LLaVA, InternVL, Qwen-VL) and closed-source (e.g., GPT-4o, GPT-5, Claude, Gemini) MLLMs.
• Robustness: The method is effective in both targeted attacks (forcing the model to output a specific description) and untargeted attacks (causing the model to lose the original semantic meaning).

4. Experimental Results

The authors evaluated MPCAttack on three datasets: ImageNet, Flickr30K, and MME.

• Performance on Open-Source Models (ImageNet):
  • Targeted Attack: MPCAttack achieved an average Attack Success Rate (ASR) of 63.33%, significantly outperforming the runner-up FOA-Attack (48.60%).
  • Untargeted Attack: MPCAttack achieved an average ASR of 92.10%, compared to 79.80% for FOA-Attack.
• Performance on Closed-Source Models (ImageNet):
  • MPCAttack demonstrated superior generalization, achieving an average ASR of 63.38% (Targeted) and 90.55% (Untargeted) against models like GPT-5 and Claude-3.5, where other methods struggled significantly.
• Ablation Studies:
  • Removing any single paradigm (e.g., removing Cross-Modal Alignment) caused a significant drop in performance, confirming the necessity of multi-paradigm collaboration.
  • Replacing the collaborative optimization (MPCO) with simple feature fusion resulted in lower ASR, proving that joint optimization is crucial for capturing global semantic relationships.
• Visualization: Visualizations showed that MPCAttack generates perturbations that are more effective at shifting the model's perception toward the target semantics compared to single-paradigm attacks.

5. Significance

• Security Implications: The paper highlights a critical vulnerability in current MLLMs: even advanced, closed-source models are susceptible to transferable attacks when the attacker leverages a multi-paradigm strategy. This suggests that current safety measures may be insufficient against sophisticated, multi-faceted adversarial inputs.
• Methodological Advancement: MPCAttack shifts the paradigm of adversarial attacks from "single-model ensemble" to "multi-paradigm collaboration." It demonstrates that leveraging the complementary strengths of different learning objectives (alignment vs. reasoning vs. self-supervision) creates a more robust and generalizable attack surface.
• Future Directions: The work underscores the need for developing more robust MLLMs that can resist attacks across diverse feature representations, rather than just defending against specific model architectures.

In conclusion, MPCAttack establishes a new state-of-the-art in adversarial attacks against MLLMs by proving that collaborative optimization across multiple learning paradigms is essential for generating highly transferable and effective adversarial examples.