NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation

1. Problem Statement

While Reinforcement Learning (RL), particularly Group Relative Policy Optimization (GRPO), has successfully enhanced Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs), applying these methods to Multimodal Large Language Models (MLLMs) reveals significant limitations:

• Limited Generalization: MLLMs trained with standard GRPO often fail to generalize beyond the training distribution. High training rewards do not consistently translate to better evaluation performance on out-of-distribution (OOD) tasks.
• Insufficient Policy Exploration: Standard GRPO relies on temperature sampling to generate multiple rollouts. However, these rollouts often converge to near-identical outputs (mode collapse), leading to insufficient exploration of the solution space.
• Lack of Process Supervision: Rule-based rewards typically judge only the final answer, leaving intermediate reasoning steps unsupervised. This encourages models to learn shortcuts and visual hallucinations that do not transfer well.
• The Off-Policy Bias: A naive attempt to inject noise into visual inputs to force exploration creates an "off-policy" scenario. The behavior policy (trained on noisy images) differs from the target policy (inference on clean images), causing distribution shifts that bias advantage estimates and corrupt policy gradients.

2. Methodology: NoisyGRPO

The authors propose NoisyGRPO, a unified RL framework that addresses the above issues through two core components without incurring significant computational overhead.

A. Noise-Injected Exploration Policy

To break the convergence of rollouts and encourage diverse exploration:

• Mechanism: During the rollout collection phase, controllable Gaussian noise is injected into the input images.
• Process: For each rollout, a noise level $n_i$ is sampled from a uniform distribution $U(0, 1)$. This noise level determines the timestep in a diffusion-style forward process applied to the image.
  • $n_i = 0$: Original clean image.
  • $n_i = 1$: Pure Gaussian noise.
• Effect: This forces the model to generate responses under varying degrees of visual degradation, encouraging it to learn robust, visually grounded reasoning rather than relying on specific visual cues or shortcuts. Crucially, noise is only applied during rollout collection; policy updates are performed on clean inputs to ensure stable learning.

B. Bayesian Advantage Estimation

To mitigate the bias introduced by the off-policy nature of noise injection, the authors formulate advantage estimation as a Bayesian inference problem:

• Formulation:
  • Prior ($\mu_i$): Derived from the injected noise level. The intuition is that if a model performs well despite high noise, the reasoning is robust. The prior reward is defined as $r^n_i = 1 - n_i$.
  • Likelihood ($\tilde{r}_i$): The observed semantic reward (based on answer correctness and format) derived from the trajectory.
  • Posterior ($\hat{r}_i$): A robust estimate of the true trajectory quality that fuses the prior and the observation.
• Uncertainty Modeling:
  • The variance of the observation ($\sigma^2_s$) is fixed (assuming stable embedding-based rewards).
  • The variance of the prior ($\sigma^2_n$) is adaptive. It is inversely related to the variance of semantic rewards within the group. If rewards are consistent across different noise levels, the prior is trusted more (lower variance). If rewards vary wildly, the prior is trusted less.
• Update Rule: The posterior mean is calculated as a weighted average:
  $$\hat{r}_i = \hat{r}^s_i + \frac{\sigma^2_s}{\sigma^2_n + \sigma^2_s} (\hat{r}^n_i - \hat{r}^s_i)$$
  This allows the model to dynamically balance reliance on the noise-based prior versus the correctness-based observation.

3. Key Contributions

1. Novel Framework: Introduction of NoisyGRPO, the first systematic multimodal RL framework that combines noise injection for exploration with Bayesian estimation for robust advantage calculation.
2. Exploration Strategy: A targeted noise injection strategy that perturbs visual inputs to diversify rollouts and implicitly favor visually grounded reasoning, solving the "mode collapse" issue in standard GRPO.
3. Bayesian Advantage Estimation: A principled method to correct the off-policy bias caused by noise injection. By treating noise level as a prior and trajectory reward as a likelihood, it computes a robust posterior advantage that guides the policy effectively.
4. Efficiency: The method introduces no additional computational cost for training a separate value model and maintains comparable training time and memory usage to standard GRPO.

4. Experimental Results

The authors evaluated NoisyGRPO on Qwen2.5-VL (3B and 7B) models across three dimensions:

• CoT Quality (MME-CoT Benchmark):
  • NoisyGRPO consistently outperformed vanilla GRPO.
  • 3B Model: Achieved a +4.4 improvement in CoT quality (F1) and significant gains in robustness.
  • 7B Model: Showed improvements in robustness, surpassing proprietary models like GPT-4o in certain robustness metrics.
• General Capability (MMStar Benchmark):
  • NoisyGRPO achieved a +3.7 gain in average performance on the 3B model and +2.6 on the 7B model compared to the base model.
  • Notable improvements were seen in Fine-grained Perception and Instance Reasoning, indicating better visual-language alignment.
• Hallucination (AMBER Benchmark):
  • Reduced hallucination rates (lower "Hal." scores) compared to GRPO, demonstrating that the method encourages more grounded responses.
• Ablation Studies:
  • Removing the Bayesian fusion (Naive NoisyGRPO) degraded performance, proving that noise injection alone is insufficient without the correction mechanism.
  • Uniform noise distribution proved superior to Gaussian or fixed noise.
  • The method showed low sensitivity to hyperparameters, indicating robustness.

5. Significance

• Scalability for Small Models: NoisyGRPO demonstrates that small-scale MLLMs (e.g., 3B parameters) can achieve reasoning capabilities comparable to or exceeding much larger models when trained with this framework. This is crucial for deploying efficient, high-performance models on edge devices.
• Robustness over Memorization: By forcing the model to reason through noisy inputs, the framework prevents the model from learning superficial shortcuts, leading to better generalization on unseen, real-world scenarios.
• Theoretical Advancement: The application of Bayesian inference to RL advantage estimation in multimodal settings offers a new paradigm for handling distribution shifts and off-policy learning, potentially applicable to other domains where input perturbation is used for exploration.

In summary, NoisyGRPO provides a robust, efficient, and theoretically grounded solution to the generalization and exploration bottlenecks currently limiting multimodal reinforcement learning, enabling MLLMs to develop more reliable and grounded Chain-of-Thought reasoning.