RL makes MLLMs see better than SFT

This paper demonstrates that Reinforcement Learning (RL) significantly outperforms Supervised Fine-Tuning (SFT) in enhancing Multimodal Large Language Models by fundamentally reshaping their vision encoders to produce stronger, more localized visual representations, leading to the proposal of a computationally efficient training framework called Preference-Instructed Vision OpTimization (PIVOT).

The Gist

The Big Idea: Teaching a Robot to "See" with a New Kind of Coach

Imagine you are training a robot to understand the world. This robot has two main parts:

1. The Brain (LLM): A super-smart language expert who knows facts, can write stories, and solve math problems.
2. The Eyes (Vision Encoder): A camera system that takes pictures and turns them into data the brain can understand.

For a long time, researchers thought the "Brain" was the most important part. They assumed if you made the Brain bigger and smarter, the robot would automatically get better at seeing. They trained these robots using a method called SFT (Supervised Fine-Tuning).

The SFT Analogy: Think of SFT like a strict teacher who says, "Here is a picture of a cat. The answer is 'cat'. Memorize this." The robot learns to repeat the correct answer, but it doesn't necessarily learn how to look at the cat's whiskers or ears deeply. It just memorizes the label.

The Discovery: The "Coach" vs. The "Teacher"

The authors of this paper asked a bold question: What if we stop just giving the robot the right answers, and instead teach it how to choose the best answer?

They introduced RL (Reinforcement Learning), specifically a method called DPO.

The RL Analogy: Imagine a sports coach. Instead of just saying, "Run this play," the coach shows the player two options:

• Option A (The Winner): A great play that scored a point.
• Option B (The Loser): A bad play that missed the goal.

The coach says, "Look at the difference. Option A worked because you looked at the defender's feet. Option B failed because you looked at the sky. Learn to focus on the feet."

The paper found that this "Coach" (RL) makes the robot's Eyes much sharper than the "Teacher" (SFT).

The Magic: How RL Changes the Eyes

When they tested this, they found something surprising. The RL method didn't just make the Brain smarter; it fundamentally rewired the Eyes.

• SFT Eyes: These eyes are a bit blurry. They see the whole picture but might miss small details. If you ask, "What is the woman holding?", an SFT-trained robot might guess "a bag" because it's a common object, even if the picture shows a stroller.
• RL Eyes: These eyes are laser-focused. They zoom in on the specific parts of the image that matter for the question. They can tell the difference between a "bag" and a "stroller" because the "Coach" taught them to look for the specific features that distinguish the two.

The Gradient Visualization: The paper even looked at the "electric signals" (gradients) inside the robot's brain.

• With SFT, the signals were scattered everywhere, like a flashlight beam shining on the whole room.
• With RL, the signals were concentrated exactly on the object being asked about, like a laser pointer hitting the specific spot.

The Result: PIVOT (The Secret Sauce)

The authors realized that this "Coach" method was so powerful they could use it to upgrade any camera system, even the best ones available today. They named this new recipe PIVOT (Preference-Instructed Vision OpTimization).

The PIVOT Analogy:
Imagine you have a standard camera (like a good smartphone camera).

• Standard Training (SFT): You take a photo, label it, and move on.
• PIVOT Training: You take a photo, show it to a human who says, "This photo is better than that one because the focus is sharper on the subject." You use that feedback to tweak the camera's lens settings.

The Shocking Outcome:
They took an older, smaller camera model (SigLIP1) and trained it with PIVOT.

• Result: This small, cheap camera, after PIVOT training, performed better than a brand new, massive, expensive camera (SigLIP2) that was trained with the old method.
• Cost: They did this with less than 1% of the computing power usually required to train these cameras. It's like turning a bicycle into a Ferrari with a simple tune-up, while the Ferrari needs a whole new engine.

Why This Matters

1. Better Vision: RL makes multimodal models (robots that see and talk) actually see better, not just talk better.
2. Efficiency: You don't need to build bigger, more expensive cameras. You just need to train the existing ones with the right "Coach" (RL/PIVOT).
3. The Future: This suggests that the future of AI vision isn't about making bigger cameras, but about teaching them how to look at the world more critically using preference-based feedback.

Summary in One Sentence

The paper proves that teaching AI to choose between "good" and "bad" answers (Reinforcement Learning) makes its eyes see details much more clearly than just teaching it to memorize the right answers (Supervised Fine-Tuning), allowing us to build super-smart vision systems that are cheaper and faster to train.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) are typically assumed to derive their capabilities primarily from their Large Language Model (LLM) backbones, leading to a research gap regarding the vision encoder. The prevailing assumption is that the vision encoder's role is static or merely a feature extractor. Furthermore, while Reinforcement Learning (RL) has revolutionized LLM alignment (e.g., via DPO), its specific impact on the visual representations within MLLMs compared to standard Supervised Fine-Tuning (SFT) remains under-explored.

The authors identify two critical questions:

1. Does RL (specifically Direct Preference Optimization, DPO) outperform SFT in MLLMs, and does this hold across different model scales?
2. How do these training strategies (SFT vs. RL) fundamentally reshape the underlying visual representations of the vision encoder?

2. Methodology

The paper employs a rigorous, controlled experimental framework to isolate the effects of training strategies on both the MLLM and its vision encoder.

A. Experimental Setup

• Architecture: The study uses the LLaVA-OneVision framework, combining various scales of Qwen2.5 LLMs (0.5B to 7B) with SigLIP2 vision encoders (B/16 to g/16).
• Training Stages:
  1. Stage 1 (Pre-training): Multimodal projector training followed by end-to-end pre-training on diverse vision-language datasets (e.g., LLaVA-OneVision-3.2M).
  2. Stage 2 (Post-training): A controlled comparison between SFT (maximizing likelihood of chosen responses) and DPO (optimizing for preference alignment using chosen vs. rejected pairs).
• Controlled Variables: The authors ensure fair comparisons by using identical dataset sizes (20K preference pairs) and model architectures, varying only the optimization objective.

B. Analysis of Visual Representations

To understand how training changes the model's "vision," the authors detach the vision encoder from the LLM after post-training and evaluate it on standalone vision tasks:

• ImageNet Classification: Linear probing to measure general visual feature quality.
• Semantic Segmentation: Patch-level probing on ADE20K to measure object localization and fine-grained understanding.
• Gradient Visualization: Using Grad-CAM to visualize where the model focuses its attention (gradients) during loss calculation, comparing SFT vs. DPO signals.
• Representation Alignment: Measuring the alignment between vision encoder features and various reference LLMs.

C. PIVOT (Preference-Instructed Vision OpTimization)

Based on findings that RL reshapes visual representations, the authors propose PIVOT. This is a recipe to evolve existing vision encoders:

1. Take a pre-trained vision encoder (e.g., CLIP, SigLIP, MAE).
2. Attach an LLM head.
3. Optimize the vision encoder using DPO (preference learning) on a small dataset (3M instruction samples + 20K preference pairs).
4. Freeze the resulting encoder and integrate it into a new MLLM.

3. Key Contributions

1. RL Outperforms SFT on Vision-Centric Tasks: The paper demonstrates that DPO-trained MLLMs significantly outperform SFT-trained counterparts, particularly on tasks requiring fine-grained visual understanding (OCR, Charts, Vision-Centric VQA), while showing comparable performance on knowledge-heavy tasks.
2. RL Reshapes Visual Representations: The most significant finding is that post-training with DPO fundamentally alters the vision encoder's internal representations. DPO-trained encoders exhibit stronger, more localized, and fine-grained visual features compared to SFT.
3. Gradient Localization: Gradient visualization reveals that DPO signals are precisely focused on question-relevant image regions, whereas SFT signals are scattered. This suggests RL provides better supervision for visual grounding.
4. PIVOT Framework: The authors introduce PIVOT, a method to upgrade existing vision encoders using RL. Remarkably, a PIVOT-enhanced smaller encoder (SigLIP2-So/16) outperforms a much larger, heavily pre-trained encoder (SigLIP2-g/16) and even next-generation encoders, despite requiring <1% of the computational cost of standard vision pre-training.

4. Key Results

• Performance Gains:
  • Vision-Centric VQA: DPO yields a +4.2% gain over SFT on OCR & Chart tasks and +2.4% on Vision-Centric tasks (with SigLIP2-L/16).
  • Scaling Laws: The advantage of DPO over SFT persists and often widens as the LLM and vision encoder scales increase.
  • Data Efficiency: DPO achieves high performance with fewer samples (e.g., 3K samples of DPO outperforms 40K samples of SFT).
• Vision-Only Benchmarks:
  • ImageNet: DPO-trained encoders show +1.83% to +1.96% higher Top-1 accuracy than SFT-trained ones.
  • Segmentation: DPO improves patch-level recall by +1.08% on CLIP-L/14 encoders.
  • Qualitative: DPO-trained encoders generate segmentation maps that align much closer to ground truth than SFT.
• PIVOT Efficacy:
  • SigLIP1 vs. SigLIP2: A PIVOT-enhanced SigLIP1-So/14 outperforms a standard SigLIP2-So/16.
  • Small vs. Large: A PIVOT-enhanced SigLIP2-So/16 (400M params) outperforms the massive SigLIP2-g/16 (1B params) in MLLM settings (55.6% vs 53.9% avg score).
  • Cost: PIVOT training takes ~18 hours on 8 H100 GPUs, compared to thousands of TPU chips for standard pre-training.

5. Significance and Impact

• Paradigm Shift: The paper challenges the "LLM-centric" view of MLLMs, proving that the vision encoder is dynamic and significantly improved by RL-based preference alignment.
• Efficiency: PIVOT offers a highly efficient path to building state-of-the-art vision backbones. It suggests that preference learning is a more effective mechanism for refining visual representations than massive-scale self-supervised pre-training alone.
• Generalizability: The benefits of RL (DPO, GRPO, PPO) over SFT are shown to generalize across different RL algorithms and model architectures, not just being an artifact of DPO.
• Future Direction: The work opens a new avenue for "Vision Encoder Optimization" via RL, suggesting that future MLLM development should prioritize preference-instructed tuning of visual backbones rather than just scaling up pre-training data.

In conclusion, the paper establishes that Reinforcement Learning does not just align the language model; it fundamentally "teaches" the vision encoder to see better, resulting in more localized, accurate, and robust visual representations that drive superior MLLM performance.