Technical Summary: ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation

1. Problem Statement

Multimodal reasoning tasks (e.g., geometry solving, chart analysis, visual question answering) require models to integrate visual inputs with multi-step reasoning. On-Policy Distillation (OPD) has emerged as a practical method to improve these capabilities by training a student model on trajectories sampled from its own policy, supervised by a teacher.

In Privileged Teacher settings, the teacher is conditioned on training-time-only signals unavailable to the student at inference. Existing Multimodal On-Policy Self-Distillation (OPSD) methods typically rely on answer-side privileges (e.g., reference answers or verified rationales). The authors identify a critical flaw in this approach:

• Train-Test Mismatch: When the teacher's supervision depends on answer-side information ($p$) that the student cannot access during inference, the student is encouraged to learn "shortcut imitation" (fitting answer-aware patterns) rather than genuine, visually grounded reasoning.
• Privilege-Induced Gap: This creates a conditional information gap where the teacher's policy depends on variables ($p$) not determined by the inference-time input ($X$), leading to suboptimal reasoning capabilities and potential performance degradation compared to the base model.

2. Methodology: ViCuR

The authors propose ViCuR (Visual Cues as Recoverable Privilege), a framework that replaces answer-side privileges with visually grounded cues.

Core Concept: Recoverable Privilege

Instead of conditioning the teacher on the answer ($R$), ViCuR conditions the teacher on a visual cue ($S$)—a description of question-relevant evidence derived directly from the input image or video (e.g., "the diagram shows two intersecting chords").

• Theoretical Justification: Under an idealized deterministic abstraction where the cue is a function of the input ($S = f(X)$), the cue introduces no additional information beyond $X$. Thus, $I(Y_t; S | X, Y_{<t}) = 0$. The teacher's supervision remains grounded in information available at inference time, mitigating the train-test mismatch.
• Practical Reality: While cues may be generated stochastically, they describe visible evidence rather than hidden answers, significantly reducing the privilege gap compared to answer-based methods.

Architecture: Sink-Based Cue Recovery

Since the student does not receive the cue text explicitly at inference, it must learn to extract relevant evidence from the visual tokens internally. ViCuR introduces a lightweight Cue Recovery Module:

• Mechanism: Inspired by the "attention sink" phenomenon, the module designates a specific token (e.g., <BOS>) as a "sink token."
• Operation: During the prefill phase (before autoregressive generation), this sink token performs dedicated cross-attention over the visual token sequence.
  • It aggregates task-relevant visual information into an internal hidden-state representation ($\hat{S}$).
  • This aggregation is guided by the distillation objective, where gradients from the teacher's supervision encourage the sink to retain visual features relevant to the question.
• Efficiency: The module adds parameters only for the cross-attention mechanism and operates solely during prefill. It introduces no autoregressive overhead during inference and preserves the standard multimodal inference interface.

Training Objective

The system uses a sampled-token on-policy distillation objective (PPO-style):

• The teacher generates supervision based on $(X, S)$.
• The student generates based on $(X, \hat{S})$, where $\hat{S}$ is the internal representation recovered by the sink token.
• The loss is computed as the advantage-weighted log-probability difference between the teacher and student. No auxiliary loss for generating cue text is required; the recovery module is trained end-to-end solely via the distillation signal.

3. Key Contributions

1. Problem Identification: The paper identifies that answer-based privileges in multimodal OPSD induce a train-test mismatch that encourages shortcut learning over visual grounding.
2. ViCuR Framework: It proposes replacing answer-side privilege with recoverable visual cues and introduces a lightweight sink-token cross-attention module to enable the student to internally aggregate this evidence without altering the inference pipeline.
3. Empirical Validation: The method is validated across seven benchmarks and two model scales (2B and 8B), demonstrating consistent improvements over both same-backbone self-distillation and stronger-teacher OPD baselines.

4. Experimental Results

Experiments were conducted using Qwen3-VL (2B and 8B) as student backbones across seven benchmarks: Vision R1-Test (in-domain), DynaMath, MathVista, WeMath, MathVerse (near-domain), and MMMU-Val, Video-MME (out-of-domain).

• vs. Answer-Based OPSD:
  • ViCuR consistently outperforms conventional answer-based OPSD.
  • 2B Student: +1.19 average improvement.
  • 8B Student: +1.24 average improvement.
  • Notably, standard OPSD often underperforms the base model (e.g., 44.88 vs. 45.42 for 2B), whereas ViCuR mitigates this degradation.
• vs. Stronger-Teacher OPD:
  • ViCuR extends effectively to settings with larger external teachers (e.g., 32B teacher for 8B student).
  • 2B Student: +0.64 average improvement over OPD baselines.
  • 8B Student: +1.08 average improvement.
  • Significant gains were observed in out-of-domain tasks (e.g., +2.30 on MMMU-Val for 2B), suggesting better transfer of reasoning capabilities.
• Ablation Studies:
  • Visual Cues: The primary source of improvement comes from replacing answer privileges with visual cues (+2.80 to +4.36 gain).
  • Recovery Module: Adding the cue recovery module provides additional gains, confirming the necessity of internal evidence aggregation.
  • Hint Leakage: ViCuR significantly reduces the explicit reproduction of "Hint" tokens in student rollouts compared to answer-based OPSD, indicating reduced reliance on privileged prompt patterns.

5. Significance and Claims

The paper argues that in multimodal on-policy distillation, the design of teacher privilege is as important as the teacher's strength.

• Grounding over Imitation: By ensuring that privileged information is recoverable from the inference-time input, ViCuR forces the student to learn visually grounded reasoning rather than fitting answer-aware shortcuts.
• Efficiency: The method achieves these gains without increasing inference latency or requiring complex auxiliary generation tasks, making it a practical upgrade for existing distillation pipelines.
• Generalizability: The approach is orthogonal to model scaling, benefiting both self-distillation and cross-scale distillation, and shows robustness across in-domain, near-domain, and out-of-domain reasoning tasks.

The authors conclude that designing privilege around inference-recoverable visual evidence is a critical, often overlooked factor in achieving high-quality multimodal reasoning in distilled models.