Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs

Technical Summary: Same Model, Different Weakness

Problem Statement
Current safety evaluations for Multimodal Large Language Models (MLLMs) often treat language and modality as independent dimensions, assuming that a model's vulnerability to jailbreak attacks is a fixed property. This paper challenges that assumption, positing that the attack surface of an MLLM is structurally language-dependent. The authors argue that alignment failures are not uniform across languages; rather, the mechanisms by which linguistic and visual attacks bypass safety filters operate through distinct pathways. Specifically, they hypothesize that Reinforcement Learning from Human Feedback (RLHF) calibrates models to recognize rhetorical attack patterns (e.g., role-play, strategic framing) primarily in English, leaving these linguistic defenses less effective or non-existent in other languages, while visual attack channels may remain consistently vulnerable or even become more potent when linguistic barriers are removed.

Methodology
The study presents the first systematic cross-lingual, multimodal red-teaming evaluation comparing US English (en-US) and Mexican Spanish (es-MX).

• Models Evaluated: Four frontier MLLMs: Claude Sonnet 4.5, GPT-5, Pixtral Large, and Qwen Omni. These were selected as next-generation successors to models evaluated in a prior English-only study by Van Doren & Ford [1].
• Adversarial Benchmark: The study utilized a fixed set of 363 unique adversarial scenarios covering three harm categories: illegal activities, disinformation, and unethical behavior. Each scenario was constructed in six attack strategies (e.g., role-play, refusal suppression) and instantiated in both text-only and multimodal (text + image) conditions.
• Linguistic Protocol: All prompts were professionally translated into Mexican Spanish by native speakers and independently reviewed to ensure fidelity to adversarial intent without cultural adaptation that might alter the attack vector.
• Data Collection: A total of 52,272 harm ratings and binary attack success judgments were collected. Nine native English-speaking and nine native Spanish-speaking annotators evaluated the model responses on a 5-point Likert scale. Annotators were blind to model identity and modality.
• Statistical Analysis: The authors employed Bayesian mixed-effects models (using brms in R) with crossed random effects for prompts and raters. This approach allowed for the decomposition of variance attributable to model, modality, language, and their interactions, avoiding the pitfalls of aggregate analysis.

Key Contributions

1. Mechanistic Evidence of Dissociation: The study provides empirical evidence that linguistic and visual alignment failures operate through distinct mechanisms. The authors demonstrate that switching the prompt language from English to Spanish causes a dissociation: linguistic attack techniques (such as role-play) become significantly less effective, while visually explicit multimodal attacks become more effective. This suggests that the prompt-language interface is a structural feature of alignment, not merely an artifact of annotator leniency.
2. Rank Reversal in Safety Rankings: The paper establishes that safety rankings are not transferable across languages. While Pixtral Large is the most vulnerable model in English conditions, Qwen Omni overtakes it as the most vulnerable in Spanish conditions. This rank reversal cannot be recovered through scalar corrections of English-condition scores, indicating that English-only benchmarks fundamentally misrepresent the risk profile of models for globally deployed users.
3. Methodological Framework: The authors demonstrate the utility of a Bayesian mixed-effects framework in safety evaluation. By jointly accounting for prompt-level and rater-level variance, this method reveals cross-lingual and cross-modal structures that aggregate analysis would miss. The authors have made their analysis scripts and data publicly available to facilitate this approach in future evaluations.

Results

• Linguistic vs. Visual Attenuation/Amplification: In Spanish (es-MX), linguistic attacks showed substantial attenuation. For instance, role-play attacks were credibly less effective (OR = 0.45) compared to English. Conversely, toxic image executions were credibly more effective in Spanish (OR = 1.58).
• Model-Specific Vulnerabilities:
  • Claude Sonnet 4.5: Remained the most resistant model across both languages and modalities, with negligible change in Attack Success Rate (ASR) between English (11.3%) and Spanish (11.4%).
  • Pixtral Large: Showed the largest drop in vulnerability when switching to Spanish (ASR dropped from 50.5% in en-US to 32.0% in es-MX), indicating a heavy reliance on English-specific linguistic alignment.
  • Qwen Omni: Exhibited a unique pattern where vulnerability increased in Spanish (ASR rose from 31.2% in en-US to 35.6% in es-MX), leading to the observed rank reversal against Pixtral Large.
• Generational Trends: Comparing these results to the prior generation of models, absolute ASRs declined in three of the four model families. However, the gaps between models did not close, and the language-dependent structure of vulnerability rankings persisted.

Significance and Claims
The paper claims that safety evaluation frameworks treating language and modality as independent dimensions fundamentally misspecify the attack surface of globally deployed MLLMs. The central finding is that the "dissociation" between linguistic and visual attack effectiveness is the predicted outcome of RLHF-based alignment being calibrated to English rhetorical patterns. Consequently, English-only benchmarks do not characterize the attack surface of models deployed to diverse populations; they only characterize the surface as seen through the lens of English-conditioned alignment.

The authors argue that pre-release safety pipelines must include native-speaker red-teaming in languages representative of the intended deployment population. They conclude that the ordering of models by safety is not a fixed property but a structural feature of current alignment practices that varies by language, necessitating a redesign of safety evaluation standards to include item-level variance decomposition rather than aggregate ASR metrics.