Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

1. Problem Statement

The development of high-quality Vision-Language Models (VLMs) relies heavily on large-scale, diverse post-training datasets. While English-speaking research has benefited from aggregating over 100 existing Visual Question Answering (VQA) datasets (e.g., FineVision with ~24M instances), this strategy is not directly transferable to low-resource languages like Japanese.

• Scarcity: Existing Japanese multimodal datasets (e.g., DEJIMA, LLM-jp-3 VILA) are significantly smaller in scale (0.4M–3.9M instances) and lack coverage in critical domains such as document understanding and chart comprehension.
• Limitations of Current Methods: Relying on proprietary models (e.g., GPT-4o) for data synthesis introduces licensing constraints, and existing Japanese pipelines often fail to cover diverse task categories necessary for robust generalization.

2. Methodology

The authors propose Jagle, a scalable pipeline for constructing a large-scale Japanese multimodal dataset from scratch, rather than merely curating existing VQA datasets. The pipeline consists of three main stages:

A. Category Definition

Based on the Eagle2 taxonomy, the authors selected five target categories relevant to Japanese language processing:

1. General VQA: Diverse natural images (people, objects, scenes).
2. Chart & Table: Extraction and interpretation of data from charts/tables.
3. Captioning & Knowledge: Image description and associated knowledge.
4. OCR QA: Question answering based on text within images.
5. Naive OCR: Direct text extraction from images in reading order.
   (Math, Science, and Text-only categories were excluded as they are less language-dependent or reserved for future work.)

B. Source Data Collection

The pipeline aggregates heterogeneous source data rather than relying on pre-labeled VQA pairs:

• Natural Images: japanese-photos, Wiki-JA (Wikipedia), and WAON (Common Crawl).
• Documents/PDFs: NDL WARP (National Diet Library), FinePDFs-Edu, and e-Gov (Japanese government portals).
• Existing Datasets: English datasets like PlotQA and TAT-QA were utilized for translation.

C. QA Generation Strategies

To generate ~9.2 million VQA pairs, four distinct strategies were employed:

1. VLM-based QA Generation: Using Qwen3-VL-235B-A22B-Instruct (a high-performance, open-weight model) to generate questions and answers from images and captions. Prompts were iteratively refined to ensure quality.
2. OCR-based Text Extraction: Using PaddleOCR-VL to extract text from images for OCR-specific tasks.
3. Text Rendering: Converting text-based datasets (e.g., JSQuAD, JSSODa) into images to create Naive OCR tasks.
4. Translation: Translating English chart/table datasets (e.g., PlotQA) into Japanese. Crucially, the Python scripts used to render the charts were translated, ensuring the text inside the images matched the Japanese QA pairs.

3. Key Contributions

• Jagle Dataset: The largest Japanese multimodal post-training dataset to date, comprising 9.2 million instances across 17 subsets and 5 task categories.
• Scalable Pipeline: A novel methodology that bypasses the need for existing VQA datasets by synthesizing data from raw images, PDFs, and translated scripts. This approach is transferable to other low-resource languages.
• Comprehensive Coverage: Unlike previous Japanese datasets, Jagle covers complex domains including document understanding, chart analysis, and multi-turn reasoning.
• Open Release: The dataset, trained models, and code are publicly released to facilitate reproducibility.

4. Experimental Results

The authors trained a 2.2B parameter VLM (Qwen3-1.7B + SigLIP2 encoder) using three settings: Jagle-only, FineVision-only (English), and a mixture of both.

• Japanese Performance:
  • The Jagle-trained model significantly outperformed the FineVision-only baseline, achieving a >20 point increase in average score across 10 Japanese benchmarks.
  • It surpassed InternVL3.5-2B and came within 5 points of the much larger Qwen3-VL-2B-Instruct, despite being trained on significantly fewer tokens (~150B vs. ~2T).
• English Performance (Cross-lingual Impact):
  • Contrary to the "curse of multilinguality," combining Jagle with FineVision improved English task performance compared to training on FineVision alone.
  • This suggests that the increased data diversity from Jagle positively impacts general multimodal reasoning, even for English tasks.
• Efficiency: The model showed no signs of saturation at 60,000 steps, indicating potential for further gains with extended training.

5. Significance

• Bridging the Language Gap: Jagle demonstrates that high-quality multilingual VLMs can be built for non-English languages without relying on scarce, pre-existing VQA datasets.
• Methodological Shift: The paper advocates for a "generate-from-scratch" approach using raw heterogeneous data (PDFs, web crawls) and advanced generation strategies, which is more sustainable for low-resource languages than the "aggregate-existing-datasets" approach used in English.
• Multilingual Synergy: The results challenge the assumption that adding non-English data degrades English performance, showing instead that diverse data sources can enhance general reasoning capabilities across languages.
• Resource Availability: By releasing the dataset and code, the authors provide a critical foundation for future research in Japanese and other low-resource multimodal AI.