INDOTABVQA: A Benchmark for Cross-Lingual Table Understanding in Bahasa Indonesia Documents

1. Problem Statement

Current Vision-Language Models (VLMs) and Visual Question Answering (VQA) benchmarks are predominantly English-centric and monolingual. They fail to adequately evaluate model performance on:

• Low-resource languages: Specifically, languages like Bahasa Indonesia, Hindi, and Arabic, which represent billions of users but are underrepresented in training data.
• Cross-lingual reasoning: The ability to understand visual content (tables) in one language (e.g., Bahasa Indonesia) while answering questions posed in a different language (e.g., English or Hindi).
• Structural complexity: Existing benchmarks often lack diversity in table layouts (bordered, borderless, colorful), making it difficult to assess a model's ability to infer structure from whitespace or visual cues.

The core gap is the inability of current VLMs to reliably process real-world documents in underrepresented languages or to align visual information across different linguistic modalities.

2. Methodology: The INDOTABVQA Benchmark

The authors introduce INDOTABVQA, a comprehensive dataset and evaluation framework designed to address these gaps.

Dataset Construction

• Source: 1,593 real-world document images from Indonesian government reports, educational records, business invoices, and public health data.
• Visual Diversity: Tables are categorized into three visual styles to test structural robustness:
  • Bordered: Traditional tables with explicit cell lines (500 images).
  • Borderless: Tables relying on whitespace and alignment (602 images).
  • Colorful: Tables using background shading or headers for grouping (491 images).
• Multilingual QA: Each document image is paired with 1,593 question-answer (QA) sets translated into four languages: Bahasa Indonesia (ID), English (EN), Hindi (HI), and Arabic (AR).
  • Annotation Process: Questions were authored in Bahasa Indonesia using controlled templates (lookup, aggregation, comparison, structural reasoning) and then translated and validated by native speakers to ensure semantic equivalence.
• Metadata: Includes table-level bounding boxes to facilitate spatial reasoning.

Evaluation Settings

The paper evaluates models across three distinct settings (illustrated in Figure 2 of the paper):

1. Zero-Shot Baseline: Pre-trained VLMs tested directly on the dataset without fine-tuning.
2. Domain-Adapted (Fine-Tuned): Models trained on a small subset of the dataset (500 images) to learn table-specific patterns.
3. Fine-Tuned + Spatial Priors: Models receive explicit table bounding box coordinates (detected via YOLOv9) as part of the input prompt. This tests whether providing spatial localization helps models focus on relevant regions, reducing ambiguity in complex layouts.

Models Evaluated

The study benchmarks a diverse range of models:

• Open-Source: Qwen2.5-VL (3B and 7B), Gemma-3 (12B), LLaMA-3.2 (11B).
• Closed-Source: GPT-4o.
• Specialized: Donut (OCR-free document transformer), used as a lower-bound baseline.

Metrics

• In-Match Accuracy: A relaxed metric checking if the normalized ground truth is a substring of the prediction (handling formatting variations).
• Semantic Textual Similarity (STS): Uses sentence embeddings (MiniLM) to measure semantic alignment between predicted and ground truth answers, crucial for multilingual evaluation.

3. Key Contributions

1. Novel Cross-Lingual Benchmark: The first dataset specifically targeting table-based VQA in Bahasa Indonesia with parallel annotations in four languages, enabling systematic evaluation of cross-lingual visual reasoning.
2. Comprehensive Baseline Analysis: A detailed failure mode analysis revealing that current SOTA models struggle significantly with low-resource languages (Hindi/Arabic) and cross-lingual alignment, even when the visual content is clear.
3. Spatial Priors & Fine-Tuning Insights: Demonstrates that explicit table localization (bounding boxes) and domain-specific fine-tuning are highly effective strategies for improving performance on specialized document tasks.

4. Key Results and Analysis

Performance Gaps

• Monolingual vs. Cross-Lingual: There is a massive performance drop when moving from Bahasa Indonesia (monolingual) to other languages.
  • Example (GPT-4o Zero-Shot): 72.2% accuracy in Bahasa drops to 44.6% (English), 26.0% (Hindi), and 21.4% (Arabic).
• Language-Specific Challenges:
  • Hindi: Consistently the lowest performing language. The authors attribute this to the Devanagari script being less common in pretraining and tokenization issues (subword tokenizers splitting meaningful morphological units).
  • Arabic: Challenges arise from Right-to-Left (RTL) text flow, which confuses models assuming Left-to-Right processing.
• Model Scale: Generally, larger models perform better, but architecture matters. Qwen2.5-VL-7B outperformed the larger Gemma-3-12B in several metrics, highlighting the importance of pretraining data quality over sheer parameter count.

Impact of Fine-Tuning and Spatial Priors

• Fine-Tuning: Fine-tuning a compact 3B model and a LoRA-finetuned 7B model on the dataset yielded significant gains:
  • 3B Model: +11.6% improvement in accuracy.
  • 7B Model: +17.8% improvement in accuracy.
• Spatial Priors: Providing explicit table coordinates improved performance by 4–7% across all models.
  • The INDOTABVQA 7B + Spatial Priors model achieved the best overall results (48.5% In-Match, 58.3% STS), outperforming the zero-shot GPT-4o and even GPT-4o with spatial priors.
  • This confirms that domain adaptation combined with spatial grounding is more effective than relying solely on massive pretraining.

Task Difficulty

• Easiest: Aggregation and Comparison questions (coarse-grained reasoning).
• Hardest: Lookup (precise cell retrieval) and Structural Reasoning (understanding headers/relationships). These tasks suffer most in cross-lingual settings.
• Visual Style: Borderless tables are the most challenging due to the lack of explicit structural cues.

5. Significance and Conclusion

INDOTABVQA addresses a critical gap in the development of inclusive, globally applicable Document AI systems.

• Research Impact: It provides a testbed for developing models that can handle the "visual-linguistic" gap in underrepresented regions (Southeast Asia, South Asia, Middle East).
• Practical Implication: The findings suggest that for specialized document tasks, targeted fine-tuning and explicit spatial localization are more effective and efficient than relying on massive, general-purpose models alone.
• Future Directions: The authors note that future work should explore finer-grained structural annotations (row/column/cell level) and expand the benchmark to include other document layouts like charts and histograms.

The dataset and code are publicly available to foster further research in cross-lingual, structure-aware document understanding.