Technical Summary: M3Kang – Evaluating Multilingual Multi-Modal Mathematical Reasoning in Vision-Language Models

1. Problem Statement

Despite the rapid advancement of Vision-Language Models (VLMs) and their demonstrated reasoning capabilities, their performance in multilingual mathematical reasoning remains underexplored. Current evaluations are heavily skewed toward high-resource languages and text-only benchmarks. Existing studies indicate that models struggle significantly in less commonly used languages and fail to accurately interpret diagrams. There is a critical lack of benchmarks that simultaneously address the intersection of multilingualism, multimodality, and mathematical reasoning. This gap limits the understanding of model robustness and hinders the development of inclusive, globally applicable AI systems. Furthermore, while some benchmarks exist for multilingual or multimodal tasks individually, few assess the combined capability, and those that do often focus on general knowledge rather than mathematics or cover only a limited set of languages.

2. Methodology

2.1 The M3Kang Dataset

The authors introduce M3Kang, the first massively multilingual, multimodal mathematical reasoning dataset for VLMs. The dataset is derived from the Kangaroo Math Competition, the world's largest mathematics contest, which engages over six million participants under 18 across 90+ countries annually.

• Source Material: The base dataset consists of 1,747 unique multiple-choice problems from the 2007–2024 editions of the competition, originally sourced from PDFs organized by the Societat Catalana de Matemàtiques.
• Scale and Diversity: The dataset includes translations into 108 culturally diverse languages, ranging from high-resource languages (e.g., English, Spanish) to low-resource languages (e.g., Tsonga, Lingala). This results in a total of 111,198 problems after filtering for quality.
• Multimodal Structure: Each problem is provided in two formats:
  1. A text-only version.
  2. An image containing the full problem (question, options, and any necessary diagrams) with the text translated into the target language.
• English Subset (M2Kang): An English-only subset is also released independently to evaluate non-multilingual, multimodal reasoning.

2.2 Dataset Construction Pipeline

The authors propose an automated pipeline to translate multimodal problems from a source language (Catalan) to 108 target languages. The process involves three stages:

1. Base Dataset Creation (Catalan): PDFs are parsed, converted to images, and segmented. Bounding boxes are manually drawn around questions to exclude Catalan-specific terms in images. An LLM-as-a-judge flags corrupted samples (e.g., incorrect bounding boxes, partial figure coverage), which are manually corrected. Duplicates are removed via textual similarity.
2. English Translation (M2Kang): The Catalan dataset is translated to English. The text is dynamically re-wrapped within the original bounding boxes to maintain multimodal alignment. An LLM-as-a-judge detects translation corruption, followed by a rigorous manual review where 42 mistranslations were removed to ensure linguistic accuracy and eliminate bias.
3. Multilingual Extension: The English dataset is extended to other languages. To ensure quality without ground-truth references for all 108 languages, the authors employ a backtranslation-based evaluation strategy.
   • Quality Metric: They use a reference-free metric based on the max chrf++ score between the source text and multiple backtranslations (using NLLB models).
   • Thresholding: Samples with a max chrf++ score below a threshold ($T = 0.625$) are discarded as mistranslations. A "high-quality" split is also provided with a stricter threshold ($T = 0.85$).

2.3 Experimental Setup

The authors benchmarked a diverse set of 16 models, including both open-source (e.g., Qwen-2.5, Gemma, Phi-4, Kimi-Thinking) and closed-source (e.g., GPT-4o, Claude-4-Sonnet, Gemini-2.5-Pro) VLMs.

• Languages: Experiments focused on 16 languages selected to represent a balanced mix of low-, mid-, and high-resource languages based on internet presence.
• Prompting: Models were prompted using Chain-of-Thought (CoT) reasoning, instructed to output reasoning traces followed by a specific answer format (A, B, C, D, or E).
• Techniques Evaluated: The study assessed standard prompting against three multilingual reasoning strategies:
  1. Model Tongue Reasoning (MTR): Translating the question to English and prompting with both the original and translated versions.
  2. CLSP: Applying MTR across multiple languages (English, Spanish, German, French) and aggregating outputs via majority voting.
  3. Activation Steering: Modifying internal model activations to align representations toward English during generation.

3. Key Results

3.1 Model Performance

• General Struggles: Despite recent advances, models still struggle with basic mathematics and diagram-based reasoning. Performance generally scales with model size and language resource availability but does not scale with grade-level difficulty.
• Top Performers: Gemini-2.5-Pro achieved the best performance across all languages and categories. Among open models, Kimi-Thinking-2506 was the strongest, outperforming GPT-4o in English but showing significant drops in low-resource languages.
• Resource Correlation: For all models except Gemini-2.5-Pro, accuracy strongly correlates with a language's Internet presence. Smaller models show a stronger correlation, indicating limited capacity to generalize across languages. Low-resource language performance for many models approaches random guessing.

3.2 Text-Only vs. Figure-Based Problems

A consistent and significant gap was observed: all capable models performed better on text-only problems than on figure-based problems, regardless of difficulty level.

• Human vs. AI Contrast: This trend is the inverse of human performance, where humans find figure-based problems easier than text-only ones.
• Implication: Current VLMs struggle to integrate visual information effectively for mathematical reasoning. The figure modality introduces complexities (spatial relationships, diagram interpretation) that current architectures do not fully capture.

3.3 Multilingual Techniques

The study evaluated whether text-based multilingual techniques transfer to multimodal settings:

• MTR Superiority: Model Tongue Reasoning (MTR) consistently achieved the strongest overall performance, substantially reducing performance disparities between low- and high-resource languages.
• CLSP and Steering: CLSP underperformed compared to MTR. Activation Steering offered modest gains only in low-resource languages and failed to match MTR's performance, even on smaller models.
• Efficiency: MTR proved effective even with smaller translator models, suggesting that in low-budget scenarios, smaller translators can be used without significant capability loss.

3.4 Human vs. Model Comparison

Using performance data from over 68,000 students from the 2024 Kangaroo competition:

• Non-Monotonic Difficulty: Models do not show a monotonic decline in performance as problem difficulty increases (based on human difficulty ratings). For instance, Kimi-Thinking-2506 ranked above the 94th percentile for levels 3–6 but only the 44th percentile for lower levels.
• Reasoning Divergence: The analysis suggests no significant correlation between the reasoning capabilities of AI models and those of young students. Models appear to rely on different problem-solving approaches, potentially rooted in pattern matching or specific training on olympiad-style geometry tasks rather than human-like step-by-step reasoning.

4. Key Contributions

1. M3Kang Dataset: A multilingual, multimodal benchmark comprising 1,747 unique problems across 108 languages (111,198 total instances), derived from the Kangaroo Math Competition.
2. Translation Pipeline: An open-source framework for automatically creating multilingual multimodal datasets from monolingual sources, featuring a backtranslation-based quality control mechanism to ensure translation integrity without ground-truth references.
3. Comprehensive Evaluation: An exhaustive benchmark of leading VLMs across languages and inference techniques, revealing that models struggle with basic math and diagrams, and that performance is heavily dependent on language resource availability.
4. Human Baseline: A direct comparison with human performance data from 68,000+ students, highlighting a fundamental disconnect between AI and human reasoning strategies in mathematical contexts.

5. Significance and Claims

The paper claims that M3Kang fills a critical gap in AI evaluation by providing a high-fidelity testbed for assessing model generalization, cross-lingual consistency, and multimodal integration in a domain requiring both linguistic precision and visual comprehension.

The authors assert that their findings reveal:

• Limitations of Current SOTA: Even top-tier models fail at basic math and diagram interpretation, with performance heavily biased toward high-resource languages.
• Effectiveness of Simple Techniques: Multilingual techniques like MTR can be effectively extended to multimodal settings, offering significant improvements over baselines without the complexity of fine-tuning or complex voting mechanisms.
• Need for New Benchmarks: The discrepancy between human and model performance (especially regarding visual vs. text-only tasks) suggests that future benchmarks must include multi-step reasoning tasks and assess intermediate reasoning steps to better capture human-like problem-solving skills.

The authors emphasize that the dataset, pipeline, and code are open-sourced to facilitate the creation of similar datasets in diverse linguistic and educational contexts, aiming to drive the development of more inclusive and robust AI systems.