EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

1. Problem Statement

Current Vision-Language Models (VLMs) lack systematic evaluation on authentic, high-stakes, multilingual document understanding. Existing benchmarks suffer from three critical limitations:

• Artificial Input Formats: Many benchmarks provide questions as clean, digitized text alongside images, or separate OCR-extracted text from the visual context. This bypasses the need for models to perform layout-aware reasoning.
• Linguistic and Script Bias: Most benchmarks are English-centric or limited to Latin scripts, failing to test models on diverse script families (e.g., Hangul, Kanji/Kana, Traditional Chinese, Devanagari) and complex typographic layouts (e.g., vertical Japanese text).
• Domain Simplification: Existing datasets often lack the visual complexity of real-world administrative documents, such as dense tables, mixed-script annotations, mathematical expressions embedded in text, and region-specific formatting conventions.

Consequently, it remains unclear whether current VLMs can handle the full pipeline of real document understanding—from perceiving complex layouts and scripts to reasoning about the content within them.

2. Methodology: The EuraGovExam Benchmark

The authors introduce EuraGovExam, a high-fidelity benchmark designed to simulate real-world civil service examinations.

Data Collection & Construction

• Sources: The dataset comprises 8,000+ high-resolution scanned multiple-choice questions sourced from official government portals in five Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union (EU).
• Domains: Questions cover 17 diverse academic and administrative domains, including law, administration, biology, mathematics, economics, and engineering.
• Construction Pipeline:
  1. Acquisition: Original PDF scans or digital versions are obtained from official agencies (e.g., UPSC in India, EPSO in the EU).
  2. Manual Segmentation: Unlike other benchmarks, no OCR extraction is performed. Trained annotators manually crop questions at the item level to preserve the original visual structure.
  3. Preservation: All content (text, tables, figures, math symbols, annotations) remains in its original language, script, font, and layout. No translation or reformatting is applied.
  4. Privacy: Personal Identifiable Information (PII) is manually removed.

Evaluation Protocol (Image-Only Setting)

The benchmark enforces a strict "Image-Only" protocol to ensure models rely solely on visual perception:

• Input: The model receives only a single image of the question. No separate text input, JSON metadata, or OCR results are provided.
• Instruction: Models receive a minimal, standardized instruction: "You are solving a multiple-choice exam question shown in the image... provide the final answer in exactly this format: The answer is X."
• Constraints: The use of external tools (OCR engines, calculators, translators) is prohibited. Models must perform end-to-end recognition and reasoning.

3. Key Contributions

1. High-Fidelity Benchmark: The first dataset to combine multilingualism (5 regions, 5 script families), multimodality (image-only input), and real-world complexity (unmodified government exam documents).
2. Factorial Analysis Design: The dataset structure (5 nations × 17 domains) enables the first systematic decomposition of performance variance into nation/script effects versus domain effects.
3. Reproducible Evaluation Kit: The authors provide a standardized evaluation script, fixed data splits, and strict inference rules to ensure fair comparison across 28 different VLMs.

4. Experimental Results

The authors evaluated 28 VLMs (14 closed-source, 14 open-source) on the benchmark.

Overall Performance

• State-of-the-Art (SOTA) Limitations: Even the best-performing model, Gemini-2.5-Pro, achieved only 86.99% accuracy.
• Open-Source Gap: Open-source models lag significantly, with the strongest (Qwen2-VL-72B) scoring 44.65%. Several models (e.g., LLaMA-3.2-11B-Vision) scored below the random baseline (23.7%), indicating stochastic guessing.
• Universal Failures: 50 questions (0.6%) were answered incorrectly by all 28 models. These failures were disproportionately concentrated in India (60% of universal failures) and involved complex non-Latin scripts combined with dense visual layouts.

Key Finding: Region Dominates Domain

The most significant finding is that nation/script factors induce 2.52× greater performance variance than domain factors.

• Cross-Regional Gaps: Within a single model, performance gaps between regions can exceed 67.9 percentage points.
  • Example: Gemini-3-Pro achieved 96.1% accuracy on Taiwanese questions but only 31.3% on Japanese questions (a 64.8% gap), despite both regions sharing the Hanzi/Kanji character family.
• Script-Specific Bottlenecks: Models struggle significantly with Japanese vertical text, mixed Kanji-Kana layouts, and furigana annotations, as well as Devanagari (India) embedded in technical diagrams.
• Implication: A model can be highly proficient in a specific domain (e.g., Philosophy) in one region but fail completely in the same domain in another region due solely to visual/script differences.

5. Significance and Implications

• Redefining Evaluation Standards: EuraGovExam demonstrates that current benchmarks (like MMLU or MMMU) likely underestimate real-world performance gaps because they control for domain but ignore script and layout complexity.
• Diagnostic Granularity: The benchmark reveals that "domain difficulty" is often a proxy for "visual/script difficulty." It identifies specific bottlenecks:
  • Japan: Visual parsing failures due to vertical layouts and furigana.
  • India: Reasoning and knowledge deficits tied to Devanagari-embedded technical content.
• Future Directions for VLMs:
  • Script-Aware Encoding: Models need layout-aware tokenization strategies for vertical text and mixed scripts.
  • Balanced Training: Training data must be augmented with underrepresented scripts (e.g., Japanese document scans, Devanagari technical content) rather than relying solely on scaling.
  • Layout as a First-Class Axis: Future benchmarks must vary layout complexity independently of language and domain.

Conclusion

EuraGovExam establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It proves that visual and linguistic properties of documents are primary determinants of model performance, often outweighing the difficulty of the subject matter itself. The benchmark serves as a critical diagnostic tool to guide the development of more robust, equitable, and globally capable vision-language models.