JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation

1. Problem Statement

While Vision-Language Models (VLMs) have advanced rapidly, reliable evaluation remains a bottleneck, particularly for Japanese language tasks. The authors identify that existing Japanese Visual Question Answering (VQA) benchmarks suffer from significantly less iterative refinement compared to their English counterparts. This lack of curation leads to several critical issues that undermine evaluation reliability:

• Ambiguity: Questions often lack a single definitive answer (e.g., open-ended or subjective queries).
• Incorrect Ground Truth: Annotated answers in the datasets are sometimes factually wrong, penalizing correct model outputs.
• Lack of Visual Grounding: Many questions can be answered using only textual cues without analyzing the image, failing to test visual understanding.
• Consequences: These noise factors distort evaluation scores, lead to misleading model comparisons, and introduce high run-to-run variance.

2. Methodology: Construction of JAMMEval

The authors introduce JAMMEval, a curated collection of Japanese benchmarks constructed through a rigorous two-round human annotation process.

• Seed Datasets: The collection is built upon seven existing Japanese VQA datasets covering diverse domains:

  • OCR (CC-OCR-JA)
  • Japanese Cultural Knowledge (JA-VLM-Bench, Heron-Bench, CVQA-JA)
  • Multi-image reasoning (JA-Multi-Image-VQA)
  • Document understanding (JDocQA)
  • Chart & Table analysis (JGraphQA)
  • Total initial instances: 1,925.

• Refinement Pipeline:

  1. Analysis: Manual inspection to identify issues (ambiguity, incorrect answers, lack of image grounding).
  2. Standardization: Converting diverse answer formats (e.g., text extraction) into a unified short-answer or multiple-choice format for consistent evaluation.
  3. Two-Round Human Annotation:
     • Round 1 (Authors): Review and refine seed datasets.
     • Round 2 (External Annotators): Catch missed issues, fix typos, and validate corrections.
  4. Handling Problematic Instances: Unlike previous works that simply filter out noisy data (which reduces sample size significantly for small Japanese datasets), JAMMEval prioritizes re-annotation and revision.
     • Actions taken: Clarifying ambiguous questions, correcting answers, replacing unanswerable instances with new QA pairs derived from the same image, or removing instances only if the image is insufficient.

• Final Statistics: The refined collection contains 1,592 instances across 7 tasks, with 1,232 unique images.

3. Key Contributions

• JAMMEval Dataset: The first systematically refined collection of Japanese VQA benchmarks, addressing the "noise" prevalent in existing resources.
• Refinement Protocol: A methodology emphasizing revision over deletion to preserve dataset size while improving quality, specifically tailored for smaller, non-English benchmark ecosystems.
• Comprehensive Evaluation: A benchmarking study of both open-weight (e.g., Qwen3-VL, InternVL3.5, Sarashina2.2) and proprietary (GPT-4o/5.1, Gemini 3 Pro) models on Japanese multimodal tasks.
• Empirical Validation of Refinement: A comparative analysis demonstrating that refined benchmarks yield more reliable, stable, and discriminative evaluation metrics.

4. Experimental Results

The authors evaluated 9 models on JAMMEval and compared performance against the original seed datasets.

A. Model Performance Highlights:

• Gemini 3 Pro: Achieved the highest overall performance (>90% accuracy on most tasks), demonstrating strong generalization.
• Qwen3-VL-8B: Led among open-weight models, notably outperforming GPT-5.1 on text-reading tasks (OCR and Documents).
• Sarashina2.2-Vision-3B: Showed superior performance on Japanese cultural tasks compared to similarly sized models, highlighting the value of Japanese-specific pretraining.

B. Impact of Refinement on Benchmark Quality:
Comparing the original datasets vs. JAMMEval revealed:

1. Increased Accuracy: Scores consistently rose after refinement (e.g., JDocQA accuracy jumped from 42.5% to 68.8%), indicating that original benchmarks systematically underestimated model capabilities due to incorrect ground truths.
2. Reduced Variance: Run-to-run standard deviation decreased significantly, making results more reproducible.
3. Improved Distinguishability: The performance gap between the best and worst models widened (e.g., CC-OCR-JA gap increased from 26.0 to 44.6), allowing for better differentiation of model capabilities.
4. Preserved Rankings: Spearman rank correlation remained near 1.0, confirming that refinement improved measurement precision without altering the relative ordering of models.

C. Error Analysis:

• Knowledge Gaps: Even top models (Gemini 3 Pro) struggled with Japanese cultural knowledge (CVQA-JA), indicating a specific area for future improvement.
• Judge Errors: A significant portion of errors in the original datasets were attributed to "Judge Errors" (ambiguity in grading), which were largely eliminated in the refined version.

5. Significance

• Reliability: JAMMEval provides a more trustworthy foundation for evaluating Japanese VLMs, eliminating the "noise" that previously led to misleading conclusions.
• Methodological Shift: The paper argues against aggressive filtering for small-language benchmarks. Instead, it demonstrates that re-annotation is a more effective strategy to maintain statistical power while ensuring data quality.
• Community Resource: By releasing the dataset and code, the authors enable future research to focus on genuine model improvements rather than battling dataset artifacts.
• Domain Specificity: The results highlight that while general VLMs are powerful, specialized Japanese models (like Sarashina) still hold advantages in cultural contexts, guiding future model development strategies.