Technical Summary: Beyond 'One Language, One Script'

Problem Statement

Current Vision-Language Models (VLMs) operate under a flawed assumption known as the "One Language, One Script" (OLOS) paradigm. This paradigm treats orthography as a deterministic function of language, assuming that a single language corresponds to a single writing system. Consequently, existing benchmarks (e.g., XM3600, MaXM) overlook billions of users who navigate the world through multi-script systems, such as Punjabi (Gurmukhi, Shahmukhi, Roman), Serbian (Cyrillic, Latin), and Kurdish (Arabic, Latin, Cyrillic).

The paper argues that this evaluation gap creates a "Script-Reality Gap." If a model's reasoning capabilities fluctuate when identical semantic content is presented in different scripts, its "multilinguality" may rely more on orthographic pattern matching than on genuine semantic grounding. This results in an equity gap where models are deemed capable in one script but unreliable in another for the same user base.

Methodology: The PuMVR Benchmark

To quantify this bias, the authors introduce PuMVR (Punjabi Multimodal Visual Reasoning), the first benchmark designed to isolate script as an independent variable in multimodal reasoning.

• Dataset Composition: PuMVR consists of 375 expert-curated, culturally grounded image-reasoning tasks. Each task exists in perfect semantic equivalence across Punjabi's three active scripts:
  1. Gurmukhi (Indic script, dominant in Indian Punjab).
  2. Shahmukhi (Perso-Arabic script, dominant in Pakistani Punjab).
  3. Roman (Latin transliteration, used by the diaspora and digital communities).
• Task Categories: The 375 instances are distributed across six reasoning categories: Visual Analogies, Cultural Object Recognition, Festival & Celebration Reasoning, Architectural & Landmark Recognition, Text-in-Image Reasoning (OCR), and Abstract Visual-Linguistic Reasoning.
• Experimental Design: The study evaluates 10 state-of-the-art VLMs (including proprietary frontier models like GPT-4o, Gemini 2.5, and Claude Sonnet 4, and open-weights models like Qwen2-VL and Llama-3.2-Vision) across five complementary investigations:
  1. Script Gap Quantification: Measuring accuracy deltas and Script Consistency Rates (SCR) across scripts.
  2. Modality Ablation: Comparing Text-Only vs. Multimodal inputs to determine if visual grounding compensates for script bias.
  3. Cross-Script Transfer: Testing few-shot learning efficiency when examples and test queries use different scripts.
  4. OCR Mismatch Robustness: Measuring performance penalties when image text and query scripts are misaligned (Script Disparity Index).
  5. Reasoning Divergence: Analyzing Chain-of-Thought (CoT) outputs to see if script choice alters reasoning pathways.

Key Contributions

1. The PuMVR Benchmark: A controlled, three-way orthographic evaluation framework that moves beyond language breadth to orthographic depth.
2. Quantification of Script Bias: The introduction of Script Consistency Rate (SCR) as a core metric for script-agnostic evaluation. The study reveals that current models exhibit substantial performance gaps, with accuracy deltas reaching up to 16% and SCR as low as 24.8% for some models.
3. Evidence of Reasoning Divergence: The paper demonstrates that script choice functions as a latent trigger, causing models to adopt divergent reasoning strategies (e.g., focusing on different visual features) for identical image-text pairs depending solely on the script used.

Key Results

• Universal Script Gap: All 10 evaluated models exhibited script-dependent performance degradation. Even frontier models like GPT-4o showed significant gaps (e.g., 4.26% delta between Gurmukhi and Shahmukhi).
• Orthographic Fragmentation: High Script Consistency Rates were rare. For instance, while GPT-4o achieved 90.93% accuracy in Gurmukhi, its SCR was only 78.13%, indicating that 12.8% of its knowledge is orthographically locked. Open-weights models performed worse, with Llama-3.2-11B-Vision showing an SCR of 27.47%.
• Visual Grounding is Additive, Not Compensatory: While visual input boosted absolute performance across all scripts (Visual Gain ranging from 7.5% to 36.1%), it did not close the script gap. The relative bias persisted, suggesting the underlying orthographic fragmentation is systematic.
• Limited Cross-Script Transfer: Reasoning patterns showed limited transferability. In few-shot settings, some models experienced catastrophic performance drops (e.g., Qwen2-VL-72B-Instruct dropped from 83.5% to 33.6% when given Gurmukhi examples for a Gurmukhi test), and Transfer Efficiency (TE) varied significantly, indicating knowledge is often script-specific rather than conceptually abstract.
• OCR and Script Mismatch: Significant performance penalties (Script Disparity Index) were observed when image text and query scripts were misaligned, particularly for Shahmukhi and in models with weaker OCR capabilities.
• CoT Divergence: Chain-of-Thought analysis revealed that structured prompting could destabilize script-specific pathways. For example, GPT-4o's Shahmukhi accuracy dropped 20.4% under CoT, while its Gurmukhi performance remained stable, suggesting Shahmukhi reasoning relies on implicit heuristics that collapse under explicit articulation.

Significance and Claims

The paper claims to expose a critical blind spot in multilingual VLM evaluation: the assumption that language breadth equates to multilingual capability. By demonstrating that state-of-the-art models are not truly multi-script, the authors challenge the OLOS paradigm.

The significance of this work lies in:

• Reframing Evaluation: Proposing Script Consistency Rate (SCR) as an essential metric to move from language counts to orthographic robustness.
• Equity in AI: Highlighting that for multi-script users (over one billion people globally), a model that succeeds in one script but fails in another is not "partially capable" but "unreliable."
• Methodological Blueprint: Providing a replicable framework for diagnosing script bias in other multi-script languages (e.g., Hindi-Urdu, Serbian, Kurdish) to achieve orthographic equity in global AI.

The authors conclude that achieving equitable AI requires moving beyond language breadth toward orthographic depth, ensuring that visual grounding and semantic reasoning are not fractured by the choice of writing system.