Technical Summary: 20/20 Vision Language Models

Problem Statement

While data curation has successfully shifted the quality–compute frontier for language models (LMs) and contrastive image-text pretraining, its specific role in Vision-Language Models (VLMs) remains under-explored. In VLM research, the training mixture is often treated merely as a volume metric ("X billion image–text pairs") rather than a primary design variable, with attention focused instead on architectural choices (encoders, connectors), resolution handling, and post-training stages (instruction tuning, RLHF). Existing VLM data curation efforts typically isolate single axes (e.g., deduplication, quality filtering, or synthetic data) or lack matched-compute baselines to prove causality. This paper asks: How far can data curation alone take VLM performance if architecture, training recipes, and compute are held fixed?

Methodology

The authors propose an end-to-end data curation pipeline applied to the MAmmoTH-VL-12M single-image subset (approx. 10M samples). The pipeline integrates four key components:

1. Multimodal Deduplication: Joint image-and-text deduplication to reduce redundancy across open-source sources.
2. Joint Filtering: Multimodal quality filters operating on both image and text signals to remove low-signal, malformed, or off-distribution samples while retaining diverse, rare-but-useful examples.
3. Mixture Design: Target-distribution matching to adjust the mixture for better coverage and balance without sacrificing the original corpus's breadth.
4. Synthetic Data: A combination of task-agnostic pipelines (to broaden coverage) and task-specific generation pipelines for selected capability families.

Experimental Setup:

• Models: Training was conducted on 1B, 2B, and 4B parameter models using Qwen3 language backbones and the SigLIP2 vision encoder.
• Training: All models were trained for 25B tokens in a single stage (joint projector and LM training) with a fixed recipe (optimizer, schedule, batch size) and compute budget.
• Baselines:
  • Matched-Compute Baseline: Trained on the uncurated MAmmoTH-VL single-image subset with identical architecture and compute.
  • Frontier References: Compared against heavily post-trained public models (Qwen3-VL, InternVL3/3.5, Qwen3.5) which utilize significantly more training compute (up to ~150×) and extensive post-training (SFT, RLHF, RLVR).
• Evaluation: Performance was measured on 20 public VLM benchmarks (11 In-Distribution, 9 Out-of-Distribution) and DatBench, a high-fidelity suite covering nine capability axes. Crucially, all training data underwent rigorous multimodal decontamination to prevent evaluation leakage.

Key Contributions

The paper demonstrates that treating pretraining data as a first-class design variable yields VLMs that are large, generalizable, reliable, and efficient, without requiring architectural changes or post-training.

1. Shift in the Cost-Quality Pareto Frontier: The curated models achieve significantly higher accuracy than the matched-compute baseline across all scales (1B, 2B, 4B).
2. Strong Out-of-Distribution (OOD) Generalization: Improvements transfer to benchmarks not used in curation design, including multi-image reasoning tasks (BLINK) despite single-image-only training.
3. Reliability and Robustness: The curation pipeline reduces variance across training seeds and maintains performance gains across varying context lengths (4k to 16k tokens).
4. Behavioral Gains Beyond Benchmarks: On open-ended queries, curated models exhibit better honesty (fewer hallucinations), specificity, conciseness, and willingness to engage with benign prompts compared to baselines and frontier models.
5. Inference Efficiency: Curated models achieve higher accuracy while generating fewer tokens, resulting in lower response FLOPs (computational cost) compared to both baselines and frontier models.

Key Results

Performance Gains

• Benchmark Lift: At the 2B scale, the curated model improved average performance by +11.7 percentage points (pp) across 20 public benchmarks and +11.3pp across the nine axes of DatBench.
• Scale Independence: Gains persisted across scales: +13.0pp at 1B, +11.7pp at 2B, and +14.0pp at 4B on the public suite.
• OOD Generalization: The 9-eval OOD average rose by +7.2pp. Notably, multi-image reasoning on BLINK improved by +3.09pp (with Visual Correspondence gaining +11.8pp) despite the model being trained exclusively on single-image data.
• Grounding Deep-Dive: The largest single gain was in Grounding (+57.1pp on DatBench), verified across multiple metrics (center accuracy, recall@k at various IoU thresholds) to be a genuine capability shift rather than a scoring artifact.

Efficiency and Cost

• Training Compute: The curated 2B model surpassed InternVL3.5-2B by 9.9pp at ~17× less training compute and closed the gap to Qwen3-VL-2B to within 1.8pp at ~87× less compute.
• Inference Cost: Curated models simultaneously raised accuracy and lowered response FLOPs. The curated 4B model matched near-frontier accuracy at 3.3× lower response FLOPs than Qwen3-VL-4B.

Reliability

• Variance Reduction: Per-capability standard deviation across training seeds dropped by ~67% (from 2.47 to 0.82pp) on DatBench.
• Context Length Robustness: The accuracy lift was preserved across context lengths of 4k, 8k, and 16k tokens.

Behavioral Improvements

On ~1,100 open-ended queries, the curated 2B model was more honest (refusing to confabulate on absent attributes) and more specific than the matched baseline. Compared to a frontier 2B reference, it was more concise (median response length 75 chars vs. 364 chars) and refused fewer benign queries.

Significance and Claims

The paper claims that data curation is a high-leverage tool for building better VLMs. By isolating data as the primary variable, the authors demonstrate that:

1. Pretraining data quality is a dominant factor: It can produce gains comparable to or exceeding those from architectural innovations and extensive post-training.
2. Efficiency is achievable: High-performance VLMs can be built with up to ~150× less training compute than current frontier models by focusing on data curation alone.
3. Generalization is inherent: High-quality pretraining data leads to robust OOD generalization and improved real-world behaviors (honesty, specificity) without explicit alignment tuning.

The authors conclude that shifting the focus from "more compute" or "new architectures" to rigorous data curation offers a Pareto-dominant path to superior VLMs, reaching near-frontier accuracy with a fraction of the resource cost.