Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages

1. Problem Statement

While Vision-Language Models (VLMs) have demonstrated strong capabilities in mathematical, scientific, and spatial reasoning on English benchmarks, there is a critical lack of evaluation regarding their performance in non-English languages, specifically within the Indian context.

• The Gap: Existing Indian-language VLM benchmarks focus on cultural understanding, OCR, or factual QA, but none isolate visual reasoning (STEM tasks).
• The Stakes: With 260 million school-age children in India studying in regional languages (Hindi, Tamil, Telugu, Bengali, Kannada, Marathi), the deployment of VLMs in EdTech raises concerns about whether these models can effectively reason about math and science in a student's native tongue.
• Hypothesis: Current models likely suffer from significant performance degradation when switching from English to Indian languages, potentially due to a reliance on English-centric reasoning chains or insufficient multilingual reasoning pretraining.

2. Methodology

The study constructs a rigorous cross-lingual audit to measure reasoning capabilities independent of visual changes.

• Dataset Construction:

  • Source: 980 visual reasoning questions were extracted from three established English benchmarks: MathVista (400 questions), ScienceQA (400 questions), and MMMU (180 questions).
  • Languages: Translated into six Indian languages representing two major language families and five scripts:
    • Indo-Aryan: Hindi (hi), Bengali (bn), Marathi (mr).
    • Dravidian: Tamil (ta), Telugu (te), Kannada (kn).
  • Translation Protocol: Used IndicTrans2 for translation, with Gemini 2.0 Flash used for cross-verification on 50 random samples per language (Inter-translator agreement: 0.79–0.84).
  • Control: Images remained identical across all languages; only the question text was translated.

• Models Evaluated:

  • 8 VLMs: Ranging from open-source (7B–32B parameters) to proprietary cloud models.
    • Proprietary: GPT-4o, Gemma 3-27B, Llama-4-Maverick.
    • Open-Source: Qwen3-VL-30B, Qwen2.5-VL-32B, Qwen2.5-VL-7B, InternVL2.5-8B, Aya-Vision-8B.
  • Total Inference Records: 68,600 (covering text-only, standard, and Chain-of-Thought ablations).

• Ablation Studies (using Qwen2.5-VL-7B):

  1. No-Image: Removed the visual input to test reliance on text.
  2. Chain-of-Thought (CoT): Added "Think step by step" prompts in the target language.

• Evaluation Metrics:

  • Accuracy on Multiple Choice Questions (MCQs) and Free-form numerical answers.
  • Cross-Lingual Consistency: Measured if the model produced the same answer across all languages for the same question.
  • Token Analysis: Measured "English Token Leak" (use of English words in non-English responses) and script confusion.

3. Key Contributions

1. First Cross-Lingual Visual Reasoning Audit for India: The paper releases the first benchmark specifically designed to test STEM visual reasoning in Indian languages, isolating language as the variable.
2. Discovery of the "Dravidian Gap": It identifies a systematic performance disparity where Dravidian languages (Tamil, Telugu, Kannada) suffer significantly larger accuracy drops compared to Indo-Aryan languages across all models.
3. CoT Backfire Effect: The study demonstrates that Chain-of-Thought prompting, often considered a reasoning booster, actually degrades performance in most Indian languages (e.g., -14.4 pp in Bengali), revealing that models cannot sustain coherent logic in these languages.
4. Resource Release: The authors released the translated benchmark, all model outputs, and evaluation code to the community.

4. Key Results

A. Overall Accuracy Drops

• Significant Penalty: Switching from English to Indian languages results in accuracy drops ranging from 9.8 to 25 percentage points (pp).
• Model Variance:
  • Gemma 3-27B showed the most robustness with the smallest average drop (9.8 pp).
  • GPT-4o achieved the highest absolute accuracy in Indian languages (43.2%) but still dropped 11.3 pp from English.
  • Open-Source Models: Suffered the most, with drops of 15–25 pp.

B. The Language Family Disparity

• Dravidian vs. Indo-Aryan: Dravidian languages consistently underperformed Indo-Aryan languages.
  • The gap was most severe for Aya-Vision-8B (13.2 pp difference), despite this model being pre-trained on 23 languages including Indian ones.
  • Gemma 3-27B was the only model with a near-zero gap (0.1 pp) between the two families.

C. Chain-of-Thought (CoT) Failure

• English: CoT had a negligible impact (-1.8 pp).
• Indian Languages: CoT caused severe degradation:
  • Bengali: -14.4 pp
  • Kannada: -11.4 pp
  • Hindi: -9.2 pp
  • Exception: Marathi saw a slight improvement (+3.3 pp).
• Implication: Models lack the fluency to generate coherent reasoning steps in Indian languages, leading to garbled logic and incorrect final answers.

D. Visual vs. Text Reliance

• Image Removal: Removing the image hurt English accuracy by 15.5 pp but hurt Indian languages less (4.9–9.7 pp).
• Interpretation: When text comprehension is weak (in Indian languages), models already under-utilize visual information. Therefore, removing the image has a smaller marginal impact.

E. Scaling Effects

• Scaling from 7B to 32B parameters (Qwen2.5-VL family) improved Indian language accuracy by 4.3 pp while slightly hurting English (-1.0 pp).
• Conclusion: Scaling narrows the gap but does not close it; larger models do not inherently solve cross-lingual reasoning transfer.

F. "English Token Leak"

• Models like Llama-4-Maverick exhibited high "English Token Leak" (32% of tokens in non-English responses were English).
• Paradoxically, this model had the highest cross-lingual consistency (80.8%), suggesting it reasons internally in English and merely translates the surface output, which works for MCQs but fails for generative tasks.

5. Significance and Implications

• Educational Risk: Deploying current VLMs in Indian regional-medium schools (mandated by the National Education Policy 2020) risks systematically disadvantaging non-English students, with accuracy penalties reaching up to 28.5 pp for Dravidian languages.
• Limitations of Pretraining: Multilingual pretraining alone (e.g., Aya-Vision-8B) is insufficient for reasoning transfer. The bottleneck is not vocabulary but reasoning capability in the target language.
• Design Recommendations:
  • Future models require multilingual reasoning fine-tuning on STEM-specific data, not just broader corpora.
  • CoT prompting should be avoided in multilingual settings unless the model is specifically tuned for non-English reasoning chains.
  • Evaluation must move beyond "cultural understanding" to rigorous reasoning audits before deployment in critical sectors like education.

The paper concludes that while VLMs are powerful, they are currently "English-centric" in their reasoning logic, and bridging this gap requires targeted architectural and training interventions rather than simple scaling or translation.