💻 computer science

Visual-Advantage On-Policy Distillation for Vision-Language Models

This paper introduces Visual-Advantage On-Policy Distillation (VA-OPD), a novel training method for Vision-Language Models that leverages token-level visual advantage signals to distinguish and prioritize vision-critical tokens during distillation, thereby significantly improving performance on mathematical reasoning and visual understanding benchmarks across various teacher model sizes.

Original authors: Ruiqi Liu, Xiaolei Lv, Gengsheng Li, Ximo Zhu, Zhiheng Wang, Zhengbo Zhang, Junkai Chen, Zhiheng Li, Bo Li, Jun Gao, Shu Wu

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruiqi Liu, Xiaolei Lv, Gengsheng Li, Ximo Zhu, Zhiheng Wang, Zhengbo Zhang, Junkai Chen, Zhiheng Li, Bo Li, Jun Gao, Shu Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Copycat" Student

Imagine you have a brilliant teacher (a large AI model) who is great at solving math problems using diagrams. You want to train a smaller, cheaper student AI to do the same thing.

Usually, you let the student try to solve a problem, and if it gets the answer right, you say, "Good job! Now, look at how the teacher solved it and copy their steps." This is called distillation.

The Catch:
The paper found a hidden flaw in this process. In a typical math problem with a picture, most of the words the AI writes are just "filler" or "scaffolding" (like saying "First, let's look at the diagram" or "The sum of angles is..."). Only a tiny few words are actually based on the picture (like reading a specific number: "130 degrees").

When the standard training method works, it treats every single word the student writes as equally important. It's like a teacher grading a student's essay where they give the same amount of attention to the word "The" as they do to the critical number "130."

The Result: The student learns to copy the words perfectly, but it doesn't actually learn to look at the picture. It becomes a "copycat" that mimics the teacher's text but ignores the visual details. If you show it a slightly different picture, it might fail because it never learned to rely on the image.

The Solution: Introducing "Visual Advantage" (VA)

The researchers invented a new way to measure how much the teacher actually needed the picture to write each specific word. They call this Visual Advantage (VA).

Think of it like a "What-If" Test:

The teacher looks at the full, high-quality picture and writes a sentence.
The teacher then looks at a blurry, pixelated version of the same picture (where the small details are gone) and tries to write the same sentence again.
The Difference: If the teacher can write the word "The" easily even with the blurry picture, that word has Low Visual Advantage (it's just language). But if the teacher gets stuck or guesses wrong on the number "130" because the blurry picture hides it, that word has High Visual Advantage.

The paper discovered that High Visual Advantage words are rare (only about 10% of the words), but they are the only ones that actually teach the student to look at the image.

How VA-OPD Works: The "Spotlight" Method

The new method, called VA-OPD, changes the training rules so the student focuses on those rare, important words. It does this in two clever ways:

1. The "Best Attempt" Bonus (Rollout-Level)

Sometimes, the student generates a few different answers to the same problem.

Old Way: Treat all attempts equally.
VA-OPD Way: It checks which attempt relied most on the picture (had the highest "Visual Advantage"). It gives that specific attempt a bonus weight, telling the student, "This is the one where you really looked at the image; pay extra attention to learning from this one."

2. The "VIP Section" (Token-Level)

Inside a single answer, the method separates the words into two groups:

The VIPs (High VA): The words that depend on the picture (like numbers read from a chart).
The Regulars (Low VA): The filler words (like "therefore," "is," "and").

Instead of averaging the learning signal across all words (which dilutes the VIPs), VA-OPD calculates the learning score for the VIPs separately. It ensures the student gets a strong "nudge" to learn the visual parts, without that signal getting drowned out by the thousands of boring filler words.

The Results: Smarter, Not Just Faster

The researchers tested this on math and visual reasoning tasks. Here is what happened:

Better Accuracy: The students trained with VA-OPD got higher scores than those trained with the old method.
Real Visual Learning: The most important finding is that the students didn't just memorize the answers. When the researchers checked, the VA-OPD students actually started relying more on the images to solve problems. Their "Visual Advantage" scores went up as they got smarter.
Scaling Up: The method worked even better when using a bigger, smarter teacher or more training data. It didn't get confused; it just got better at spotting the important visual clues.

Summary Analogy

Imagine you are teaching a child to identify fruits in a basket.

Standard Training: You show them a picture of an apple and say, "This is a red, round fruit that grows on trees." The child memorizes the sentence "red, round, grows on trees" but doesn't actually learn to recognize the shape or color of the apple.
VA-OPD Training: You realize the child only needs to pay attention to the words "red" and "round" because those are the parts that prove it's an apple. You ignore the words "grows on trees" (which could apply to pears too) and give the child a special reward every time they correctly identify the "red" and "round" parts.
Outcome: The child learns to actually see the apple, not just recite the description.

In short: This paper fixes a blind spot in AI training by teaching smaller models to stop copying text and start actually looking at the pictures.

Technical Summary: Visual-Advantage On-Policy Distillation for Vision-Language Models

1. Problem Statement

On-policy knowledge distillation has proven effective for compressing Large Language Models (LLMs) by training a student to match a teacher's output distribution on its own generated rollouts. However, applying this paradigm to Vision-Language Models (VLMs) reveals a critical mismatch: standard distillation treats all tokens uniformly, failing to distinguish between language scaffolding and tokens that genuinely depend on fine-grained visual evidence.

The authors observe that in typical VLM rollouts, the vast majority of tokens serve as language templates or reasoning structures, while only a small minority are determined by visual details (e.g., specific numbers in a diagram). Standard Kullback-Leibler (KL) divergence minimization averages gradients uniformly across all tokens. Consequently, the strong supervision signal present in the few visual-critical tokens is diluted by the abundant surrounding language tokens. As a result, while the student's output accuracy may improve, its underlying reliance on visual input does not strengthen; the student learns to mimic the teacher's surface-level text without actually learning to attend to the image.

2. Methodology: Visual-Advantage On-Policy Distillation (VA-OPD)

To address this dilution, the authors propose Visual-Advantage On-Policy Distillation (VA-OPD), a method that explicitly identifies and prioritizes tokens requiring fine-grained visual detail.

2.1 Visual Advantage (VA)

The core diagnostic tool is Visual Advantage (VA), a token-level metric quantifying the teacher's reliance on visual detail. For each token $y_t$ in a student-generated rollout, VA is calculated as the non-negatively rectified log-probability difference between the teacher's score with the original image ( $v$ ) and a degraded image ( $\tilde{v}$ ) where fine-grained details (e.g., OCR, small symbols) are destroyed via downsampling:
$a_t = \max(\log p_T(y_t | v, q, y_{<t}) - \log p_T(y_t | \tilde{v}, q, y_{<t}), 0)$
High VA indicates the token prediction depends heavily on fine-grained visual detail, while low VA suggests the token relies on language priors or robust global image features.

2.2 Key Observations

Empirical analysis of student rollouts reveals two critical properties:

Sparsity: VA is heavy-tailed; the top 10% of tokens carry approximately 93% of the total VA mass.
Supervision Concentration: Masking the KL loss on high-VA tokens causes a significant drop in accuracy, whereas masking low-VA or random tokens has negligible effect. This confirms that the visual supervision signal is concentrated in a small minority of tokens.

2.3 The VA-OPD Objective

VA-OPD modifies the standard on-policy distillation objective by applying VA at two granularities to prevent signal dilution:

Rollout-Level Reweighting: For a set of $K$ sibling rollouts generated for the same prompt, each rollout is assigned a soft weight based on its trajectory-averaged VA relative to its siblings. Rollouts with higher visual reliance contribute more to the gradient.
Token-Level Grouped KL: Within each rollout, tokens are sorted by VA and split into a High-VA group (top fraction, e.g., 20%) and a Low-VA group (remainder). The per-token KL loss is averaged separately within each group before being summed. This ensures the gradient contribution from the critical high-VA tokens is not scaled down by the total sequence length $T$ .

The final loss function combines these components:
$\mathcal{L}_{\text{VA-OPD}} = \mathbb{E}_{x \sim D} \left[ \sum_{k=1}^K w^{(k)} \left( \lambda \cdot \frac{1}{|V^{(k)}|} \sum_{t \in V^{(k)}} \text{KL}_t + (1-\lambda) \cdot \frac{1}{|L^{(k)}|} \sum_{t \in L^{(k)}} \text{KL}_t \right) \right]$
where $w^{(k)}$ is the rollout weight, and $\lambda$ balances the high-VA and low-VA group contributions.

3. Key Contributions

Identification of the Dilution Problem: The authors demonstrate that standard on-policy distillation for VLMs fails to strengthen visual reliance because uniform KL weighting dilutes the concentrated visual supervision signal found in a small minority of tokens.
Introduction of Visual Advantage (VA): They propose VA as a teacher-relative counterfactual metric to quantify token-level visual dependency without requiring extra annotations or reward models.
VA-OPD Framework: They introduce a distillation objective that utilizes VA at two levels (rollout reweighting and token-level grouped KL) to concentrate the learning signal on vision-critical positions.
Empirical Validation: The method is validated across three teacher sizes (4B, 8B, 32B), two datasets (Geometry3K, ViRL39K), and eight benchmarks, showing consistent improvements over standard baselines.

4. Experimental Results

The authors evaluated VA-OPD on the Qwen3-VL family (8B teacher, 2B student) trained on Geometry3K, with generalization tests on ViRL39K and different teacher sizes.

Performance Gains: VA-OPD outperformed Standard OPD on all eight benchmarks.
- Mathematical Reasoning: Average improvement of +2.9 points.
- Visual Understanding: Average improvement of +1.5 points.
- Specific Benchmarks: Notable gains were observed on HallusionBench (+2.5) and AI2D (+2.4), tasks heavily reliant on fine-grained visual detail. Benchmarks allowing text-only shortcuts (e.g., MMStar) showed smaller gains, consistent with the method's focus on visual dependency.
Scaling Laws: The performance gain of VA-OPD over Standard OPD increased monotonically with both teacher size (from 4B to 32B) and dataset scale (from Geometry3K to ViRL39K), suggesting the factors compound consistently.
Visual Reliance: During training, the mean VA of student rollouts rose in lockstep with accuracy for VA-OPD, whereas Standard OPD showed rising accuracy with stagnant VA. This indicates VA-OPD achieves genuine visual reliance rather than mere output mimicry.
Efficiency: VA-OPD achieved the final accuracy of Standard OPD approximately 3.0x faster in wall-clock time and reached a higher performance ceiling at fixed time budgets.

5. Significance and Claims

The paper claims that VA-OPD represents the first method to embed per-token reliance on fine-grained visual detail directly into the on-policy distillation target for VLMs. Its significance lies in:

Mechanism: It solves the "dilution" problem by explicitly separating visual-critical tokens from language scaffolding during the distillation process.
Efficiency: It requires no additional annotations, reward models, or changes to the student's generation process, adding only one extra teacher forward pass per rollout (with degraded visual detail).
Generalizability: The method improves performance across varying model scales and data volumes, suggesting it addresses a fundamental limitation of current VLM distillation strategies rather than a specific dataset artifact.
Qualitative Shift: Unlike inference-time methods that modify decoding, VA-OPD modifies the training target to ensure the student learns to "read" the image, as evidenced by the emergence of VA hotspots on numerical values and diagram features in qualitative visualizations.

The authors remain modest regarding limitations, noting that VA is a proxy for visual dependency based on teacher counterfactuals rather than a direct measurement of internal attention mechanisms, and that gains are most pronounced on vision-intensive tasks.