JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding

1. Problem Statement

While Vision-Language Models (VLMs) are increasingly replacing modular OCR pipelines for end-to-end text understanding, evaluating their performance on Japanese scene text remains a significant challenge. Existing benchmarks suffer from three main limitations:

• Lack of Diagnostic Depth: Multilingual benchmarks often prioritize language breadth over language-specific diagnostics, failing to disentangle whether a model's failure stems from character recognition (reading the text) or reasoning (understanding the content).
• Script Complexity: Japanese text involves a complex mix of scripts (Kanji, Hiragana, Katakana, and Latin alphanumerics), frequent vertical writing, and a character inventory far larger than the Latin alphabet.
• Data Gaps: Existing Japanese datasets focus heavily on scanned documents or isolated characters, leaving "in-the-wild" scene text (e.g., street signs, receipts, handwritten notes) underexplored.

Consequently, there is no existing resource capable of fine-grained diagnosis of where and why VLMs fail on Japanese text in real-world scenarios.

2. Methodology: The JaWildText Benchmark

The authors introduce JaWildText, a diagnostic benchmark comprising 3,241 evaluation instances derived from 2,961 newly captured images in Japan. The dataset contains 1.12 million annotated characters spanning 3,643 unique character types.

The benchmark is designed around three complementary tasks that vary in visual organization, output format, and writing style to isolate specific failure modes:

A. Dense Scene Text Visual Question Answering (Dense STVQA)

• Goal: Evaluate reasoning over multiple pieces of visual text evidence in cluttered environments (signboards, posters, product packages).
• Data: 1,025 question-answer pairs across 745 images.
• Design: Questions require cross-referencing multiple text regions. Crucially, each question is linked to evidence regions (minimal text regions required to answer). This allows the system to determine if a model failed because it couldn't read the evidence (Recognition Error) or because it couldn't synthesize the answer (Reasoning Error).
• Question Types: Compositional retrieval, counting, calculation, and spatial reasoning.

B. Receipt Key Information Extraction (Receipt KIE)

• Goal: Test layout-aware structured extraction from mobile-captured receipts.
• Data: 1,151 unique receipt images.
• Design: Focuses on rigid columnar layouts, mixed full/half-width characters, and domain-specific abbreviations. It evaluates the extraction of specific fields (e.g., store name, date, total amount, tax) and line-item tuples.
• Challenge: Tests the ability to handle geometric distortions (folds, creases) and missing fields without hallucination.

C. Handwriting OCR

• Goal: Evaluate page-level transcription of multi-sentence handwritten Japanese text.
• Data: 1,065 instances across 1,065 images.
• Design: Covers diverse media (lined paper, whiteboards, tablets) and writing directions (horizontal/vertical).
• Annotation Principle: Ground truth reflects exactly what is visually present, including writer errors (misspellings, incomplete strokes), ensuring the model is evaluated on visual recognition fidelity rather than error correction.

3. Key Contributions

1. First Dedicated Japanese Benchmark: JaWildText is the first benchmark specifically designed to evaluate VLMs on Japanese scene text across three distinct, real-world task types.
2. Disentangled Evaluation: By linking evidence regions to QA tasks and using structured field extraction, the benchmark separates recognition failures from reasoning failures, providing a diagnostic capability absent in previous resources.
3. Comprehensive Baselines: The authors benchmarked 14 open-weight VLMs (including Qwen3-VL, InternVL3.5, Gemma3, Phi-4-Multimodal, and the Japanese-centric Sarashina2.2-Vision), establishing reproducible baselines and quantifying performance gaps.
4. Fine-Grained Error Analysis: The paper provides a taxonomy of errors (Recognition, Reasoning, Format) and a script-level breakdown (Kanji vs. Kana vs. ASCII), revealing specific bottlenecks for different model families.

4. Experimental Results

The authors evaluated 14 models, with Qwen3-VL-8B achieving the best overall score of 0.64.

• Overall Performance: Even the best model struggles, with an average score of 0.64, indicating that robust Japanese scene text understanding is far from solved.
• Task-Specific Performance:
  • Dense STVQA: Shows the widest performance spread (0.008 to 0.62), effectively differentiating model capabilities.
  • Receipt KIE: Models struggle significantly with spatial reasoning over document layouts (e.g., extracting store_name and store_address), with field-level accuracies as low as 0.14 for store names, despite high accuracy on constrained fields like time.
  • Handwriting OCR: Most models achieve a baseline >0.60, but performance plateaus around 0.80 even for larger models.
• Scaling Laws:
  • Performance generally improves with parameter count, but architecture and training data composition matter more than scale alone. For instance, Qwen3-VL-8B outperforms InternVL3.5-38B.
  • Sarashina2.2-Vision-3B (trained on Japanese data) matches the 38B InternVL model on STVQA but fails to generalize to KIE or Handwriting tasks, suggesting Japanese-centric training benefits are task-dependent.
• Error Analysis:
  • Recognition is the Dominant Bottleneck: For most models, recognition errors account for the majority of failures in STVQA.
  • Script-Specific Issues: Kanji exhibits the highest Character Error Rate (CER) due to the large character inventory. Models like InternVL3.5 struggle with visually similar Katakana, while Gemma3 struggles with Kanji and ASCII letters.
  • Format Errors: Some models (e.g., Phi-4-Multimodal) fail primarily due to an inability to follow output formatting constraints.

5. Significance and Conclusion

JaWildText provides a critical diagnostic tool for the research community, moving beyond aggregated accuracy to reveal the specific mechanisms of failure in Japanese VLMs.

• Key Insight: The primary bottleneck for Japanese scene text understanding remains character recognition, particularly for Kanji, rather than high-level reasoning.
• Implication: Improving Japanese VLMs requires targeted interventions in the recognition stage (e.g., better handling of large character sets and mixed scripts) rather than just scaling up reasoning capabilities.
• Future Work: The authors hope this benchmark encourages similar fine-grained, script-aware diagnostic evaluations for other typologically diverse languages.

The dataset and evaluation code are set to be released under the Apache License 2.0 to facilitate further research.