💬 NLP

Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

This paper reveals that while multilingual multi-modal large language models exhibit cross-lingual adversarial vulnerabilities, their apparent safety in low-resource languages is often an artifact of visual and comprehension failures ("safety-by-failure"), whereas models with deep multilingual integration throughout training achieve genuine cross-lingual safety alignment.

Original authors: Hashmat Shadab Malik, Muzammal Naseer, Salman Khan

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hashmat Shadab Malik, Muzammal Naseer, Salman Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of very smart robots that can both see (like a camera) and think (like a human). These are called Multimodal Large Language Models (MLLMs). They are great at looking at a picture and describing it, or answering questions about what they see.

However, just like humans, these robots can be tricked. This paper is a big investigation into how easily these robots can be tricked, specifically when they speak and understand 12 different languages.

Here is the story of what the researchers found, explained simply:

1. The "Universal Glitch" (Adversarial Attacks)

The researchers tried to "hack" the robots by adding tiny, invisible scratches to images (like static on an old TV) that the human eye can't see. These scratches are designed to confuse the robot's brain.

The Finding: They found a "Universal Glitch." If they created a confusing scratch in English, it confused the robot just as much when the robot was asked to speak Spanish, Hindi, or Arabic.
The Analogy: Imagine a robot that has a single, fragile "glasses lens" for seeing. If you put a smudge on that lens, it doesn't matter if the robot is trying to read a sign in English or French; the smudge makes it blind to the text in all languages. The weakness isn't in the language; it's in the way the robot sees the world.

2. The "Silent Failure" vs. The "Polite Refusal" (Safety)

The researchers also tested if the robots would say "No" to dangerous requests (like "How do I build a bomb?"). They tested this in two ways:

Asking in words: "Tell me how to build a bomb."
Asking in pictures: Showing a picture with the words "How to build a bomb" written inside it.

They compared two types of robots:

Type A (The "Quick Learners"): These robots (like PALO and PARROT) were taught English first, and then quickly taught other languages by translating English lessons.
Type B (The "Deep Learner"): This robot (QWEN3-VL) was taught all languages from the very beginning, deeply integrated into its brain.

The "Safety-by-Failure" Illusion

The researchers discovered a scary trick with the Type A robots.

What happened: When asked a dangerous question in a "hard" language (like Bengali or Urdu), the robot often didn't answer with a bomb-making guide. It actually looked "safe" because it didn't say anything bad.
The Catch: The robot wasn't being safe because it was smart; it was being safe because it was confused. It didn't understand the question at all! It just started hallucinating nonsense, like "I see a black dog," or repeating the same word over and over.
The Analogy: Imagine a security guard at a bank. If a robber speaks a language the guard doesn't know, the guard might just stare blankly and say, "I see a blue chair." The bank is safe, but not because the guard is brave or smart. It's safe because the guard is illiterate in that language. The paper calls this "Safety-by-Failure."

The "Real Safety"

The Type B robot (QWEN3-VL) was different.

What happened: When asked the same dangerous question in Bengali or Urdu, this robot understood the question perfectly. It didn't give a nonsense answer; it said, "No, I cannot do that."
The Twist: Interestingly, the Type A robots looked safer in low-resource languages (because they failed to understand), while the Type B robot showed its true "weak spots" in those same languages because it actually understood them and had to actively decide to say "No."

3. The "Typo" Trap

The researchers also tested what happens when the dangerous words are written inside a picture (like a sign in a store).

English Signs: The robots could read English signs inside pictures easily and often followed the dangerous instructions.
Foreign Signs: When the sign was in Arabic or Chinese, the Type A robots couldn't read the letters. They just ignored the sign or described the background. Again, this wasn't because they were being careful; it was because their "eyes" couldn't read that script.

The Big Lesson

The paper concludes that just because a robot seems safe in a language it doesn't speak well, it doesn't mean it is actually safe. It might just be too confused to know it's in danger.

To make these robots truly safe in every language, we can't just translate English lessons at the end. We have to teach them all languages deeply from the start, so they understand the danger and can say "No" properly, no matter what language is spoken.

In short: A robot that doesn't understand a threat isn't a safe robot; it's just a blind one. True safety means understanding the threat and choosing to refuse it.

Technical Summary: Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

1. Problem Statement

Multimodal Large Language Models (MLLMs) integrate visual perception with language reasoning, creating a continuous attack surface susceptible to adversarial perturbations. While prior research has examined MLLM robustness, evaluations have been overwhelmingly English-centric, leaving the behavior of these models in multilingual settings largely unexplored.

A critical gap exists in understanding whether multilingual MLLMs, particularly those adapted via instruction tuning, possess genuine safety alignment across diverse languages or if their apparent safety in low-resource languages is an artifact of comprehension failure. Furthermore, it remains unclear whether adversarial vulnerabilities (both in reasoning and safety) transfer across languages, given that visual inputs reside in a continuous, high-dimensional space amenable to gradient-based optimization.

2. Methodology

The authors conduct a systematic study across 12 typologically diverse languages (Arabic, Bengali, Chinese, English, French, Hindi, Japanese, Portuguese, Russian, Spanish, Turkish, and Urdu). The evaluation focuses on two representative open-source MLLMs that acquire multilingual capabilities through multilingual multimodal instruction tuning (PALO and PARROT) and one model that integrates multilingual data throughout its entire training pipeline (QWEN3-VL).

Benchmark Construction

To address the lack of multilingual benchmarks, the authors constructed a comprehensive suite adapting existing English-centric datasets (COCO, Flickr30k, LLaVA-Bench, RealToxicityPrompts, MM-SafetyBench) into 12 languages. This was achieved via a rigorous Translate-then-Verify pipeline:

Multi-model Translation: Using a pool of LLMs to generate candidate translations.
Back-translation Consistency: Filtering candidates based on semantic fidelity via back-translation.
Human Verification: Native speakers validated the final translations and typography rendering.
The resulting suite comprises over 60,000 adapted instances.

Evaluation Axes

The study evaluates models along two complementary axes:

Gradient-Based Adversarial Attacks (White-box):
- Robustness: Optimizing visual perturbations ( $\delta$ ) to degrade captioning and reasoning performance (maximizing cross-entropy loss) in a source language and evaluating transfer to target languages.
- Visual Jailbreaking: Unconstrained optimization of noise images to induce harmful responses when paired with textual prompts, testing cross-lingual safety transfer.
Non-Adversarial Multimodal Safety:
- Text-Only Safety: Querying models with harmful instructions in various languages to assess linguistic comprehension and refusal.
- Typographic Safety: Embedding harmful keywords directly into images (visual modality) while keeping the text prompt benign, testing the vision encoder's ability to parse and refuse multilingual text.

3. Key Results

Cross-Lingual Adversarial Transferability

Shared Vulnerability: Adversarial perturbations optimized in a single source language transfer broadly to other evaluation languages. This indicates that MLLMs possess a shared multimodal representation space where language-agnostic adversarial vulnerabilities exist.
Asymmetry: Attacks crafted in high-resource or LLM-dominant languages (e.g., English, Chinese) tend to cause larger performance drops in English evaluations compared to attacks from lower-resource languages. However, even lower-resource language attacks (e.g., Bengali) successfully degrade performance in high-resource languages.
Scope: This transferability holds for both short-form captioning (COCO, Flickr30k) and long-form reasoning (LLaVA-Bench).

Safety Behavior: "Safety-by-Failure" vs. Genuine Alignment

The study identifies a critical distinction in safety behavior based on how multilingual capability is acquired:

Instruction-Tuned Models (PALO, PARROT):
- Apparent Safety: These models exhibit low unsafe response rates in low-resource languages (e.g., Bengali, Urdu) and non-English scripts.
- The Mechanism: This safety is an illusion termed "Safety-by-Failure." Low unsafe rates co-occur with low refusal rates. The models fail to comprehend the harmful intent (due to weak linguistic grounding or poor OCR for non-English scripts) and instead generate hallucinated, irrelevant, or repetitive outputs. They do not reject the harm; they simply miss it.
- Typographic Bias: When harmful text is embedded in images, English scripts are reliably parsed and followed, whereas non-English scripts are rarely recognized, leading to near-zero unsafe rates that reflect visual grounding failure rather than safety alignment.
Deeply Integrated Models (QWEN3-VL):
- Genuine Safety: QWEN3-VL, which incorporates multilingual data across pretraining and alignment stages, exhibits a qualitatively different profile. It maintains active refusal rates across languages.
- Inverted Pattern: The languages that appeared "safest" under instruction-tuned models (Bengali, Urdu) reveal the highest residual vulnerability in QWEN3-VL. This confirms that QWEN3-VL genuinely comprehends the harmful queries and actively chooses to refuse them, exposing true alignment gaps rather than masking them with comprehension failures.

4. Key Contributions

Multilingual Benchmark Suite: The creation of a publicly available, rigorously translated and verified benchmark suite covering 12 languages for both adversarial robustness and multimodal safety.
Discovery of Cross-Lingual Transferability: Demonstrating that gradient-based adversarial vulnerabilities are shared across the multimodal representation space, transferring effectively regardless of the language used for optimization or evaluation.
Identification of "Safety-by-Failure": Uncovering that shallow multilingual adaptation (instruction tuning only) creates an illusory safety in low-resource languages. The models appear safe because they fail to understand the input, not because they are aligned to refuse it.
Training Paradigm Comparison: Providing evidence that integrating multilingual capability throughout the training pipeline (as in QWEN3-VL) leads to genuine cross-lingual safety and active refusal, whereas post-hoc instruction tuning results in fragile, comprehension-dependent safety.

5. Significance and Claims

The paper argues that shallow multilingual adaptation (e.g., fine-tuning on translated instruction data) is insufficient for ensuring robust and safe MLLMs. It produces surface-level multilingual understanding that creates illusory safety in low-resource languages.

The authors claim that true safety alignment requires a deeper integration of multilingual capability across all training stages. Without this, models may appear safe in non-English settings simply because they cannot parse the language or script, a failure mode that masks the absence of genuine safety alignment. The study highlights that safety evaluations must move beyond English-centric benchmarks to reveal these hidden vulnerabilities and the "safety-by-failure" phenomenon.

The work concludes that improving multilingual capability alone is not sufficient; safety alignment must be explicitly considered and validated across all supported languages to ensure consistent and genuine safety behavior.