🤖 machine learning

Stage-1 Controls the Entropy Regime, Not the Outcome

This study demonstrates that while Stage-1 warm-start methods (SFT vs. OPD) significantly influence the initial entropy regime and answer diversity of vision-language models, they do not substantially alter the final in-domain performance or provide a clear downstream advantage after Stage-2 reinforcement learning.

Original authors: Jianxiong Shen

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianxiong Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a smart robot (a Vision-Language Model) to solve complex geometry puzzles. The paper investigates the best way to "warm up" this robot before sending it into a high-stakes training camp called Reinforcement Learning (RL).

There are two main ways to warm up the robot:

SFT (Supervised Fine-Tuning): You show the robot a textbook of perfect answers written by a super-smart teacher and say, "Memorize these exactly."
OPD (On-Policy Distillation): You let the robot try to solve problems itself, but whenever it gets stuck or makes a mistake, the super-smart teacher whispers the right next step. The robot learns by mimicking the teacher's process, not just the final answer.

The researchers wanted to know: Does the way we warm up the robot change the final result?

Here is the breakdown of their findings, using simple analogies:

1. The "Final Score" is Surprisingly Similar

Think of the final exam (solving geometry problems) as a race. The researchers tried three different warm-up methods.

The Finding: No matter which warm-up they used, the robot finished the race in almost the exact same spot (around 53–54% accuracy).
The Analogy: It's like three different runners starting from slightly different spots on a track. By the time they reach the finish line, they are all bunched together in a tiny cluster. The warm-up didn't give anyone a massive head start that lasted until the end.

2. The "Forgetting" Myth

Some people worry that if you teach a robot too much specific math (SFT), it will forget how to do other things (like general math puzzles).

The Finding: This only happened if you trained the robot too long or with the wrong instructions. If you stopped training at the right time, the robot didn't forget anything.
The Analogy: It's like studying for a specific history test. If you cram for 10 hours, you might forget your own name. But if you study for just 1 hour, you remember the test and your name. The problem wasn't the studying method; it was studying for too long.

3. The Real Difference: "Entropy" (The Robot's Mood)

This is the paper's biggest discovery. While the final scores were the same, the internal state of the robots was very different.

SFT Robots: These became very confident and rigid. They knew the answer and stuck to it. Their "entropy" (a measure of randomness or variety in their thinking) was very low. They were like a robot that says, "I know the answer is 4, and I will never say 5."
OPD Robots: These remained curious and varied. They kept a few different possibilities in mind. Their "entropy" was much higher. They were like a robot saying, "The answer is probably 4, but maybe 5 or 6 is possible too."
The Analogy: Imagine a chef.
- The SFT chef is a robot that only cooks one specific recipe perfectly.
- The OPD chef is a robot that knows the main recipe but also remembers the teacher's "maybe try a pinch of salt" or "maybe add more spice." The OPD chef has a wider menu of ideas.

4. Does Being "Curious" Help?

The researchers asked: "Does having that extra variety (high entropy) help the robot solve more problems?"

At the Start (Before the final training): Yes! The OPD robot (the curious one) could generate many different correct answers if you gave it 16 tries. It was better at exploring options.
After the Final Training (RL): No. Once the robot went through the high-stakes training camp, that initial curiosity vanished. Both the rigid SFT robot and the curious OPD robot ended up solving the same number of problems.
On New Puzzles (Out-of-Domain): The curiosity didn't help the robot solve new types of math problems either. The advantage disappeared completely.

The Bottom Line

The paper concludes that the choice of warm-up (SFT vs. OPD) is like choosing between a strict drill sergeant and a flexible coach.

What it controls: It controls the robot's "personality" (how rigid or flexible its thinking is) during the early stages.
What it doesn't control: It does not guarantee a better final score or better performance on new, unseen problems.

The Takeaway: If you want a robot that thinks with more variety right now, use the flexible coach (OPD). But don't expect that variety to automatically turn into a higher final score after the robot finishes its training. The "warm-up" sets the mood, but the "training camp" (RL) is what actually determines the final result.

Technical Summary: Stage-1 Controls the Entropy Regime, Not the Outcome

Problem Statement
Two-stage post-training pipelines for Vision-Language Models (VLMs)—consisting of a Stage-1 warm-start (Supervised Fine-Tuning, SFT, or On-Policy Distillation, OPD) followed by Stage-2 Reinforcement Learning (RL)—are increasingly standard. A prevailing assumption in the community is that the choice of warm-start significantly dictates the final RL outcome. Specifically, proponents argue that SFT may induce catastrophic forgetting of base capabilities, while OPD (training a student on its own rollouts under a teacher's token distribution) provides a superior initialization for RL. This paper interrogates these claims within a small-data regime (≈1–2k examples) using the Qwen2.5-VL-7B model, aiming to determine what Stage-1 actually controls and whether it meaningfully alters the downstream endpoint.

Methodology
The study employs a controlled experimental setup comparing three distinct warm-start strategies followed by closely matched Stage-2 GRPO (Group Relative Policy Optimization) recipes:

OPD: On-policy distillation from a same-modality 72B Qwen2.5-VL teacher over all 2,101 Geometry3K training prompts.
SFTID: In-domain optimized SFT using the teacher's correct traces (via rejection sampling) with a system prompt.
SFTOOD: Out-of-domain (OOD) optimized SFT using the same teacher traces but without a system prompt and early-stopped at 100 steps.

Key Distinctions in Setup:

Teacher Consistency: All warm-starts derive from the same 72B teacher, eliminating teacher-identity confounds.
Data Coverage Confound: A deliberate difference exists in data coverage. SFT traces are derived from rejection sampling, retaining only the 807 problems the teacher solves within 4 samples. OPD trains on all 2,101 prompts, including 1,294 problems the teacher never solves. The authors treat this as a known setting difference and analyze its impact on entropy.
Evaluation: Models are evaluated in-domain (Geometry3K internal validation) and out-of-domain (MathVista testmini). Metrics include policy entropy, pass@k (with $k \in \{1, 4, 8, 16\}$ ), and answer diversity.

Key Findings

1. Forgetting is Recipe-Dependent (F1)
The study challenges the notion that small-data SFT inherently causes catastrophic forgetting.

An over-trained SFT variant (200 steps, no system prompt) showed a significant drop in OOD MathVista performance (−9.5 points).
However, a matched-recipe, early-stopped SFT (100 steps) actually improved OOD performance by +2.1 points over the base model.
Conclusion: The observed forgetting is not an intrinsic property of SFT but a consequence of specific recipe choices (e.g., training duration, prompt engineering).

2. In-Domain Endpoints Occupy a Narrow Band (F2)
Despite different warm-starts, the final in-domain performance on Geometry3K converges to a narrow range.

All three warm-starts, after GRPO, reached a best internal validation score between 53% and 54%.
This aligns with recent specialized methods, suggesting limited "headroom" for separating warm-starts on this specific benchmark rather than a universal saturation of the task.

3. Stage-1 Controls the Entropy Regime, Not the Outcome (F3)
The most distinct and reliable difference between the methods is the entropy regime carried into RL.

Entropy Gap: OPD enters the RL phase with substantially higher policy entropy (0.10–0.16 initially) compared to SFT variants (0.015–0.04). This gap persists through the available trajectories, with OPD remaining higher even late in training.
In-Domain Initialization Advantage: At the initialization stage (before RL), the high-entropy OPD model demonstrates higher exploration potential. While greedy accuracy (pass@1) is comparable or slightly lower, OPD achieves a significant advantage in pass@16 (+2.0 to +5.2 points over SFT baselines) and higher answer diversity.
Vanishing Payoff: This advantage is strictly bounded:
- Post-RL: After Stage-2 GRPO, the pass@16 advantage disappears. The final endpoints for all models are within 1.1 points of each other, with confidence intervals crossing zero.
- OOD Transfer: On MathVista, the entropy ordering disappears entirely. All six models (three initializations × two stages) fall within a 1.2-point range at pass@16.

Significance and Claims
The paper explicitly frames its contribution as a bounded empirical characterization rather than a new method or a leaderboard record.

What Stage-1 Controls: The primary finding is that Stage-1 strongly associates with the entropy regime of the policy entering RL. OPD maintains a high-entropy state, while SFT collapses to a low-entropy, deterministic state.
What Stage-1 Does Not Control: The study finds no evidence that the choice of warm-start (OPD vs. SFT) determines the final RL endpoint or OOD generalization in this small-data setup. The "downstream payoff" of the entropy advantage is small, localized to the in-domain initialization, and does not survive the RL optimization process.
Caveats: The authors acknowledge a confound: the entropy gap may be partially driven by OPD seeing the "hard tail" of problems (1,294 unsolved by the teacher) that SFT never sees. While the objective-level intuition (reverse-KL vs. cross-entropy) supports the entropy mechanism, the paper does not claim causality without a data-aligned control (training OPD on the same 807 problems as SFT).

Conclusion
In this small-data VLM study, the choice of Stage-1 warm-start is a strong determinant of the entropy regime but not the final outcome. While OPD offers a higher-entropy initialization with better in-domain exploration (pass@16), this does not translate to a superior RL endpoint or OOD performance. The paper concludes that entropy is the clearest observed difference, but it is not evidence that OPD is a universally better RL warm-start.