Technical Summary: VMMU – A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

Problem Statement

While Vision-Language Models (VLMs) have demonstrated significant progress on English-centric multimodal benchmarks, their capabilities in low-resource languages remain underexplored, particularly when tasks require the joint integration of language understanding, visual grounding, and reasoning. Existing Vietnamese benchmarks either lack multimodal questions entirely or rely on "weakly multimodal" formats (e.g., text-only screenshots) where the image serves merely as a container for text. Consequently, there is a critical gap in evaluating how VLMs handle genuine multimodal questions in Vietnamese, where non-text visual evidence (diagrams, charts, maps, traffic scenes) is essential to solving the problem and must be integrated with rendered Vietnamese text.

Methodology

Benchmark Construction (VMMU)

The authors introduce VMMU, a benchmark comprising 2,548 carefully curated multiple-choice questions across 7 domains: Mathematics, Physics, Chemistry, Biology, Geography, Driving Test, and IQ Test.

• Data Sources: Questions are sourced from official government documents (Ministry of Education and Transportation), national academic tests (High School Graduation Examination), and curated public repositories. All usage complies with Article 25 of Vietnam's IP Law for non-commercial research.
• Curation Pipeline: A semi-automated pipeline renders PDFs into images, uses Tesseract OCR and contour analysis to detect question boundaries and visual elements, and filters out text-only items. The final dataset undergoes triple manual verification by native Vietnamese speakers to ensure modality labels, crop accuracy, and answer correctness.
• Question Representations: VMMU distinguishes between four input types but primarily evaluates (D) Integrated-MM, where questions, options, and visual evidence are interleaved in a single image. It also provides (C) Split-MM variants (text separated from visual evidence) to diagnose failure modes.

Evaluation Protocol

The study evaluates 5 state-of-the-art (SOTA) proprietary VLMs: Gemini-2.5-Flash, Sonnet-4.0, GPT-4.1, o3, and Gemini-3.0-Pro. The evaluation is structured into several ablation studies:

1. OCR Reliability: Assessing embedded text transcription accuracy using human-AI verified ground truth.
2. Integrated Performance: Measuring end-to-end accuracy on the original Integrated-MM images.
3. Split-MM Analysis: Re-evaluating models with text provided as plain input and visual evidence as a separate crop to isolate grounding/reasoning from OCR/layout distractions.
4. Translation Analysis: Translating Split-MM text to English to test if language barriers are the primary bottleneck.
5. Visual Evidence Removal: Removing visual crops entirely to measure reliance on text-only priors and heuristics.

Key Results

1. OCR is Not the Bottleneck

Contrary to assumptions that low-resource language performance is limited by text recognition, the evaluated models demonstrated strong Vietnamese OCR capabilities.

• Metrics: Mean BLEU of 89.01%, F1 of 94.30%, and low Character Error Rates (CER: 6.59%).
• Implication: Errors in downstream tasks are primarily driven by multimodal grounding (locating relevant visual evidence) and reasoning, not by the inability to read Vietnamese text.

2. Reasoning Capabilities Drive Performance

There is a significant performance gap between "non-thinking" models (standard inference) and "thinking" models (models with explicit reasoning chains).

• Overall Accuracy: The mean accuracy across all models in the Integrated-MM setting is 66.76%.
• Model Comparison: Non-thinking models achieved 50.58%–71.34% accuracy, while thinking models (o3, Gemini-3.0-Pro) achieved 73.98%–86.33%.
• Domain Difficulty: IQ Tests (abstract visual reasoning) and Traffic Situations were the most challenging domains, while Tables and Traffic Signs were comparatively easier.

3. Input Format Significantly Impacts Accuracy

• Split-MM Improvement: Separating text from visual evidence (Split-MM) improved accuracy for all models by an average of +5.98%. The gain was most pronounced for non-thinking models (+8.56%), suggesting that dense layouts in Integrated-MM images introduce distractions that hinder grounding.
• Translation Detriment: Translating Vietnamese text to English reduced accuracy by an average of 2.30%. This indicates that English translation does not serve as a shortcut and may introduce a "mixed-language grounding" problem where English instructions fail to align with Vietnamese visual labels.

4. Reliance on Priors

When visual evidence was removed entirely, model accuracy dropped significantly (average -21.27%) but remained well above random chance (51.47% vs. 25.92% baseline). This suggests that models often rely on text-only priors, exam heuristics, or memorized patterns to guess answers when visual grounding is absent.

Significance and Contributions

The paper claims VMMU addresses a critical gap in multimodal evaluation by providing a benchmark specifically designed for genuine multimodal reasoning in Vietnamese. Its primary contributions are:

1. Diagnostic Insight: It demonstrates that for Vietnamese VLMs, the primary failure mode is not OCR but the integration of text and visual evidence and the subsequent reasoning steps.
2. Methodological Clarity: By distinguishing between Integrated and Split inputs, the authors show that current models struggle with the complexity of dense, interleaved layouts, and that "thinking" mechanisms significantly mitigate this.
3. Language Specificity: The results refute the hypothesis that translating to English improves performance, highlighting the necessity of native-language evaluation for tasks involving language-specific visual conventions (e.g., traffic signs, administrative forms).
4. Community Resource: VMMU serves as a foundation for future research into Vietnamese-centric instruction tuning and the development of datasets that target grounding and reasoning skills beyond English-centric resources.

The authors conclude that while proprietary models show promise, reliable multimodal reasoning in Vietnamese requires overcoming challenges in joint text-vision reasoning under dense layouts and improving multimodal grounding rather than simply improving OCR or language translation capabilities.