Technical Summary: DeltaPrompts – Escaping the Zero-Delta Trap in Multimodal Distillation

Problem Statement

Knowledge distillation has become a pivotal mechanism for endowing compact Vision-Language Models (VLMs) with strong reasoning capabilities by leveraging stronger teacher models. However, current practices rely heavily on off-the-shelf datasets (e.g., ChartQA, SciVQA) where prompts are selected via simple heuristics or aggregated from existing pools. The authors identify a critical inefficiency in this approach: a significant portion of these prompts are "zero-delta."

A prompt is defined as zero-delta when the teacher and student models induce the exact same answer distribution for a given input. In such cases, the distillation signal provides minimal learning value because the student is not exposed to a capability gap it needs to bridge. The paper reveals that up to 69% of prompts in standard chart and document reasoning datasets are effectively zero-delta. Training on these prompts causes student performance to saturate rapidly, regardless of data scale. Furthermore, simply filtering existing datasets to retain only non-zero-delta prompts is suboptimal, as it drastically reduces the diversity of the training data (by up to 29.4%), leading to a trade-off where divergence is gained at the expense of generalization.

Methodology

To address the zero-delta trap, the authors propose a shift from passive data curation to active prompt synthesis driven by the principle of Answer Divergence ($\Delta$).

1. Defining Answer Divergence

Instead of measuring sequence-level divergence (which is noisy and computationally intractable due to variable-length outputs), the authors define Answer Divergence over the discrete space of final answers.

• Metric: $\Delta(x) = D_{KL}(P_\theta(\cdot|x) \parallel P_T(\cdot|x))$, where $P_\theta$ and $P_T$ are the empirical answer distributions of the student and teacher, respectively.
• Estimation: The distributions are approximated by sampling $K=16$ trajectories from both models. An LLM judge (Qwen3-32B) groups semantically equivalent answers to handle surface-level variations, allowing for robust computation of the divergence even for open-ended answers.

2. The DELTAPROMPTS Synthesis Pipeline

The authors propose a staged synthesis pipeline that repurposes existing datasets as seeds rather than direct training resources. The pipeline consists of three stages:

• Stage 1: Seed-guided Generation: Existing image-prompt pairs are used as few-shot examples to prompt the teacher model to generate novel, harder problems for new images. This expands the candidate pool.
• Stage 2: Skill-guided Generation: To explicitly target student failure modes, the pipeline extracts "skills" by contrasting teacher trajectories (correct) against student trajectories (incorrect) for prompts with non-zero divergence. These skills (e.g., "interpolating logarithmic scales") are then used to condition the data generator, forcing it to create prompts that require these specific missing capabilities.
• Stage 3: Zero-Delta Rejection: All generated prompts are evaluated for answer divergence. Prompts where the teacher and student agree (zero-delta) are rejected. Only prompts with high teacher consistency and non-zero divergence are retained.

This process results in DELTAPROMPTS, a dataset of 200k synthetic, high-divergence reasoning problems covering chart/document reasoning and high-resolution, perception-centric reasoning.

Key Contributions

1. Identification of the Zero-Delta Trap: The paper empirically demonstrates that standard multimodal reasoning datasets are dominated by zero-delta prompts, which lead to early saturation in student performance during distillation.
2. Answer Divergence as a Principled Metric: The authors introduce answer divergence as a tractable and critical metric for quantifying the teacher-student capability gap, showing it is a stronger predictor of distillation success than raw data scale or diversity alone.
3. A Two-Stage Synthesis Pipeline: A novel method for generating high-quality training data that actively seeks out student failure modes via skill-guided generation, rather than passively filtering existing data.
4. DELTAPROMPTS Dataset: A large-scale, diverse dataset of 200k prompts specifically engineered to maximize answer divergence, outperforming existing mixtures in both divergence and diversity metrics.

Experimental Results

The authors evaluate DELTAPROMPTS across three distinct settings using Qwen3-VL and GLM-4 model families:

• On-Policy Distillation: When distilling a student (Qwen3-VL-8B-Thinking) from a teacher (Qwen3-VL-235B-Thinking), DELTAPROMPTS yields up to 15% relative improvement across 10 benchmarks (including ChartQA, InfoVQA, and RealWorldQA), even when the base student is already a highly optimized reasoning model.
• Cross-Family Transfer: The dataset generalizes to a different model family (GLM-4.1V-9B-Thinking) without regenerating the data, simply by re-filtering for non-zero divergence under the new pair.
• Off-Policy Fine-Tuning (SFT): When used for standard supervised fine-tuning on a non-reasoning model (Qwen3-VL-8B-Instruct), DELTAPROMPTS substantially outperforms SFT on existing data mixtures and shows better retention on out-of-domain tasks (Math, Logic, STEM) compared to standard SFT.
• Ablation Studies:
  • Divergence vs. Diversity: Controlling for diversity, training on delta prompts ($\Delta > 0$) continues to improve performance with data size, while zero-delta prompts ($\Delta = 0$) saturate immediately.
  • Consistency Filtering: Prompts where the teacher is inconsistent (hard to solve even for the teacher) degrade performance, validating the need for consistency-based filtering in the synthesis pipeline.

Significance and Claims

The paper claims that the central bottleneck in multimodal distillation is not the aggregation of more prompts or trajectories, but the construction of prompts where the teacher and student meaningfully disagree.

By shifting focus from data quantity to divergence quality, the authors demonstrate that targeted prompt synthesis can yield substantial gains with significantly less data compared to heuristic aggregation of existing datasets. The work suggests that answer divergence is a fundamental property for effective scaling in distillation, offering a path to train capable compact models more efficiently. The authors position DELTAPROMPTS not just as a dataset, but as a proof-of-concept for a more principled, data-centric approach to distillation that actively targets the specific capability gaps of the student model.