From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for training Multimodal Large Language Models (MLLMs), particularly using algorithms like Group Relative Policy Optimization (GRPO). However, RLVR relies heavily on high-quality, verifiable labels to compute rewards. In real-world scenarios, datasets often contain significant annotation noise (incorrect labels).

The paper identifies three critical failure modes when applying standard RLVR methods to noisy data:

1. Overfitting to Noise: Models tend to overfit to incorrect labels, leading to poor generalization.
2. Suppression of Diversity: Standard entropy minimization (often used to boost confidence) suppresses the response diversity required by GRPO. GRPO relies on group-wise advantage estimation, which needs diverse outputs within a group to calculate meaningful reward signals. If diversity collapses, the advantage estimation becomes unreliable.
3. Ineffectiveness of Static Entropy Strategies: Existing methods that either purely maximize entropy (to encourage exploration) or purely minimize entropy (to encourage exploitation) fail to balance the trade-off. Pure maximization prevents convergence, while pure minimization traps the model in sub-optimal deterministic behaviors when facing noisy labels.

2. Methodology: Two-Stage Entropy-Guided RLVR

The authors propose a Two-Stage Entropy Optimization framework that dynamically guides the model from Exploration to Exploitation during the training process.

Core Concept: Token-Level Entropy

Instead of sequence-level entropy, the method utilizes token-level entropy ($\mathcal{H}_{token}$), which measures the uncertainty of the model's probability distribution at each generation step.

• Entropy Loss ($\mathcal{L}_{entropy}$): Defined as the negative expected average token-level entropy across sampled responses.

The Two-Stage Schedule

The total loss function combines the standard GRPO loss ($\mathcal{L}_{GRPO}$) with a dynamic entropy regularization term:
$$\mathcal{L}_{total} = \mathcal{L}_{GRPO} + \lambda(\tau) \mathcal{L}_{entropy}$$
Where $\tau$ is the training step and $\lambda(\tau)$ is a time-dependent coefficient.

1. Stage 1: Exploration (Entropy Maximization)

   • Timing: Early training phase (e.g., first 80% of steps).
   • Mechanism: $\lambda(\tau) = +\lambda_{max}$. The model is penalized for low entropy, effectively maximizing token-level entropy.
   • Goal: Encourage diverse outputs and broad exploration. This acts as a regularizer to prevent premature convergence to noisy labels and ensures sufficient intra-group variation for GRPO's advantage estimation.

2. Stage 2: Exploitation (Entropy Minimization)

   • Timing: Late training phase (after a switching point $\tau_{switch}$).
   • Mechanism: $\lambda(\tau) = -\lambda_{min}$. The sign flips, effectively minimizing token-level entropy.
   • Goal: Encourage confident, deterministic outputs. This consolidates the knowledge acquired during the exploration phase and refines prediction accuracy.

3. Key Contributions

1. Systematic Evaluation of Noise: The authors conducted comprehensive experiments across multiple dimensions: varying noise levels (0% to 100% incorrect labels), diverse model architectures (Qwen2-VL, Qwen2.5-VL, InternVL), and multiple tasks (GUI grounding, fine-grained classification, open-vocabulary object detection).
2. Identification of Limitations: They demonstrated that while standard GRPO has inherent robustness due to group-wise normalization, adding static entropy minimization or maximization alone is suboptimal under noisy supervision.
3. Novel Two-Stage Strategy: The proposal of a phased entropy schedule that transitions from exploration to exploitation. This approach mitigates overfitting to noise while ensuring the model eventually converges to a high-confidence policy.
4. Superior Performance: The method outperforms representative baselines, including external-signal methods (LLM-as-a-Judge), internal-signal methods (random/format rewards), and existing entropy-based methods.

4. Experimental Results

The method was evaluated on ScreenSpot (GUI grounding), Pets37 (fine-grained classification), and COCO (open-vocabulary object detection).

• Robustness to Noise:

  • On the ScreenSpot task with 50% noisy labels, the proposed method achieved 80.2% accuracy, compared to 76.2% for standard GRPO and 77.4% for GRPO with entropy minimization.
  • Under 100% noisy labels (pure noise), the method achieved 75.8%, significantly outperforming standard GRPO (71.0%) and entropy-minimization baselines (73.2%).
  • The method showed consistent gains across all noise levels, with improvements ranging from 4.0% to 13.0% over the base model.

• Model Scalability:

  • The approach proved effective across different model sizes (2B, 3B, 7B parameters) and architectures (Qwen and InternVL families).
  • Larger backbones (e.g., Qwen2-VL-7B) showed significant improvements (e.g., +8.6% at 50% noise), suggesting the method scales well.

• Qualitative Analysis:

  • Entropy Minimization: Caused the model to collapse into a single, confident reasoning path early on, propagating noisy rewards and leading to incorrect coordinates.
  • Entropy Maximization: Generated diverse reasoning paths but failed to consolidate, resulting in lower accuracy.
  • Two-Stage Method: Maintained diverse reasoning traces during the exploration phase to resist noise, then converged to a coherent, correct path during the exploitation phase.

• Out-of-Distribution (OOD) Generalization:

  • The method demonstrated superior generalization on unseen benchmarks (ScreenSpot-Pro, OS-World-G, MMBench-GUI L2), achieving the highest accuracy in most configurations, particularly when trained with a mix of clean and noisy data.

5. Significance

This paper addresses a critical bottleneck in the deployment of RLVR for MLLMs: the reliance on perfect labels. By introducing a dynamic entropy schedule, the authors provide a mechanism that allows models to learn effectively from imperfect, noisy data without sacrificing the diversity required for stable reinforcement learning.

The findings suggest that entropy is not a static property to be minimized or maximized, but a dynamic resource that should be managed throughout the training lifecycle. This "Exploration-to-Exploitation" paradigm offers a robust, task-agnostic solution for training MLLMs in real-world scenarios where high-quality annotations are scarce or noisy.