🤖 machine learning

Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training

This paper proposes that Reinforcement Learning (RL) outperforms Supervised Fine-Tuning (SFT) in VLM generalization because it implicitly prioritizes medium-difficulty data, leading the authors to introduce "Difficulty-Curated SFT" (DC-SFT), a more efficient method that achieves superior out-of-distribution performance by explicitly filtering for optimal sample difficulty.

Original authors: Aojun Lu, Tao Feng, Hangjie Yuan, Wei Li, Yanan Sun

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aojun Lu, Tao Feng, Hangjie Yuan, Wei Li, Yanan Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a student to become a world-class chef. You have two different teaching methods, and you want to know which one will help the student succeed when they are thrown into a real, unpredictable professional kitchen.

The Two Teachers: SFT vs. RL

1. The "SFT" Teacher (The Perfectionist Textbook Teacher):
This teacher gives the student a massive stack of recipe books. Every single recipe is "correct." The teacher says, "Memorize these exactly. If the book says add two grams of salt, you add exactly two grams."

The Problem: The student becomes a master at following those specific books. But if they walk into a kitchen where the ingredients are slightly different, or the stove is hotter, or the recipe is written in a different style, they panic. They "overfit"—they learned the books, not the art of cooking.

2. The "RL" Teacher (The "Trial and Error" Coach):
This teacher doesn't give books. Instead, they give the student ingredients and say, "Try to make a soup. I'll taste it and give you a score."

The Benefit: The student learns by doing. They try something, it tastes bad, they adjust. They try something else, it tastes great, they do more of that. This student is much better at handling a real kitchen because they’ve learned how to adapt.

The Big Discovery: The "Goldilocks" Secret

For a long time, scientists wondered: "Why is the RL Coach so much better at helping students adapt than the SFT Textbook Teacher?"

This paper discovered the answer isn't about the teaching style—it’s about the difficulty of the recipes.

The researchers realized that the RL Coach has a "secret filter." When the student is given a task:

If it's too easy (the soup is already perfect), the coach doesn't say much. There's no room to learn.
If it's too hard (the student is trying to cook a 5-course meal with no heat), the student fails every time. The coach gives a low score, but since the student fails every attempt, they don't learn how to improve.
If it's "Just Right" (Medium Difficulty): This is the sweet spot. The student gets it right sometimes and wrong sometimes. The coach provides feedback that says, "You were close! Try this instead."

The RL Coach naturally spends all their time on the "Just Right" tasks, while the SFT Teacher wastes time forcing the student to memorize "Impossible" tasks that actually confuse the student and ruin their intuition.

The Solution: DC-SFT (The "Smart Curriculum")

The authors created a new method called DC-SFT (Difficulty-Curated SFT).

Instead of being a "Textbook Teacher" who forces the student to read every single book (including the ones that are too hard and confusing), the DC-SFT teacher curates the library. They look at the books beforehand and say: "We are going to throw away the books that are too easy (boring) and the books that are impossibly hard (confusing). We will only train the student on the books that are 'Just Right'."

The Result?

By simply being picky about the data, this "Smart Teacher" (DC-SFT) actually beat the "Trial and Error Coach" (RL)!

Why is this a big deal?

It's Faster: You don't have to spend weeks watching the student fail and try again (RL is slow). You just give them the right books and go.
It's More Stable: The student doesn't have "bad days" where they suddenly forget everything (RL can be unpredictable).
It's Smarter: The student becomes better at "reasoning"—they don't just memorize; they actually understand the logic, making them ready for the real world.

Technical Summary: Why Does RL Generalize Better Than SFT?

1. Problem Statement

In the post-training of large-scale Vision-Language Models (VLMs), a consistent performance gap is observed: models trained via Reinforcement Learning (RL) demonstrate significantly better out-of-distribution (OOD) generalization compared to those trained via Supervised Fine-Tuning (SFT). While existing literature often attributes this to RL's optimization objective (exploration and reward feedback), this paper investigates whether the gap is actually driven by the data distribution the models learn from.

2. Methodology & Hypothesis

The Data-Centric Hypothesis:
The authors propose that RL's advantage is not necessarily algorithmic, but rather acts as an implicit data filter. They categorize training samples into three difficulty levels based on model rollouts:

Easy: All generated responses are correct (high, uniform reward).
Medium-difficulty: A mixture of correct and incorrect responses (high reward variance).
Hard: All generated responses are incorrect (low, uniform reward).

In RL (specifically using the GRPO algorithm), easy and hard samples yield uniform rewards, resulting in a relative advantage ( $A_k$ ) of zero, which leads to negligible gradient updates. Consequently, RL implicitly concentrates its learning on medium-difficulty samples. Conversely, standard SFT applies uniform updates to all data, meaning "hard" samples—which produce disproportionately large gradient norms—can dominate the optimization and cause overfitting, thereby degrading OOD performance.

Proposed Method: Difficulty-Curated SFT (DC-SFT)
To test this, the authors introduce DC-SFT, a method that explicitly filters the training set before SFT:

SFT-M: Mimics RL by training exclusively on medium-difficulty samples.
SFT-EM: Combines both easy and medium-difficulty samples (removing only the "hard" samples) to maximize robustness.

3. Key Contributions

New Theoretical Perspective: Provides a data-centric explanation for the RL-SFT generalization gap, shifting the focus from optimization objectives to implicit data selection.
Empirical Validation: Systematically proves that training on hard samples is the primary driver of OOD degradation in SFT.
DC-SFT Framework: Introduces a simple, efficient, and stable alternative to RL that achieves superior generalization.
Gradient Analysis: Demonstrates through gradient norm visualization that hard samples exert an outsized, detrimental influence on the model's optimization trajectory.

4. Experimental Results

The authors evaluated their methods using Qwen2.5-VL (3B & 7B) and MiniCPM-V-4 across image classification and visual grounding tasks.

Superior Generalization: DC-SFT (specifically the SFT-EM variant) consistently outperformed both standard SFT and RL-based GRPO on OOD benchmarks (e.g., ImageNet-R/A and Ref-L4/Lisa).
Efficiency & Stability: DC-SFT is significantly more efficient than RL. On ImageNet, DC-SFT achieved a 4.9× speedup; on RefCOCO, a 3.2× speedup. Furthermore, unlike GRPO, which shows sharp performance drops during training, DC-SFT maintains high stability.
Reasoning Capabilities: On complex multimodal reasoning benchmarks (MMMU, MathVista, etc.), the SFT-M variant achieved the highest average scores, outperforming both SFT and RL.
Robustness to Hard Data: Ablation studies showed that even a small inclusion (5%) of hard samples significantly degrades OOD accuracy, confirming their detrimental impact.

5. Significance

This work is significant because it challenges the necessity of complex RL pipelines for achieving high-quality VLM post-training. By demonstrating that data curation (filtering hard samples) is more critical than the optimization algorithm (RL vs. SFT) for generalization, the paper provides a more practical, computationally efficient, and stable pathway for developing robust, reasoning-capable Vision-Language Models.