Technical Summary: MARS – Mono-Anchored Advantage Normalization for Multi-Source Visual Reasoning

Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced visual reasoning, existing methods struggle when processing multi-source inputs (e.g., RGB, infrared, depth, multi-view). Current approaches typically treat multi-source data as a simple accumulation of information, operating under the assumption that "seeing more means knowing more." However, this paper identifies a critical limitation: when sources differ significantly in physical properties and semantics, naive integration can introduce interference or noise rather than information gain.

In scenarios where one source holds a dominant, reliable signal (e.g., infrared in low-light conditions) while others are degraded or irrelevant, standard RLVR frameworks often fail to distinguish between beneficial fusion and detrimental conflict. Consequently, multi-source reasoning performance can degrade below the level of strong mono-source reasoning. Existing frameworks lack explicit mechanisms to model the dynamic interaction between sources or to quantify whether integrating an additional source yields a positive information gain relative to a mono-source baseline.

Methodology: MARS

To address these challenges, the authors propose MARS (Mono-Anchored Advantage Normalization for Multi-Source Reasoning). This framework treats each visual modality as an independent information source and explicitly models the information gain introduced by multi-source fusion.

Core Mechanism

The central innovation of MARS is the use of mono-source rewards as dynamic anchors to normalize the advantages of multi-source rollouts during RLVR training.

1. Dual Rollout Generation: For a given training instance (question $q$ and multiple images $i$), the policy model $\pi_\theta$ generates two types of trajectories:

   • Multi-source rollouts ($G_{multi}$): The model processes all available images jointly.
   • Mono-source rollouts ($G_{mono}$): The model processes each image individually paired with the text input.

2. Hybrid Advantage Normalization: Instead of calculating advantage statistics solely from multi-source trajectories, MARS computes a hybrid mean and standard deviation using the union of both mono-source and multi-source rewards ($G_{multi} \cup G_{mono}$). The advantage $A_j$ for a multi-source trajectory is calculated as:
   $$A_j^{hy} = \frac{r(q, i, o_j^{multi}) - \text{mean}(G_{multi} \cup G_{mono})}{\text{std}(G_{multi} \cup G_{mono})}$$

3. Adaptive Regulation:

   • Conflict Suppression: If a specific source is dominant and multi-source fusion introduces noise (resulting in lower rewards than the mono-source anchor), the normalization statistics shift to suppress the conflicting multi-source signal, guiding the model toward the reliable mono-source behavior.
   • Promotion Emphasis: If multi-source fusion yields a reward higher than the mono-source baseline (indicating genuine information gain), the algorithm encourages exploration beyond the mono-source cues, effectively quantifying the "multi-source information gain."

Theoretical Foundation

The authors provide theoretical analysis to support the method:

• Unbiasedness: The proposed algorithm guarantees that the expectation of the gradient estimation remains equivalent to standard on-policy optimization, ensuring no bias is introduced.
• Gradient Decomposition: The gradient update is shown to be equivalent to maximizing the standard multi-source reward plus a regularization term based on the Multi-Source Information Gain ($\Delta IG$). This term represents the difference between the expected rewards of multi-source and mono-source trajectories. A negative $\Delta IG$ indicates modality conflict, while a positive value indicates mutual promotion.

Key Contributions

1. Identification of a Core Limitation: The paper reveals that existing multi-source visual reasoning can systematically degrade performance due to unmodeled modality interference, challenging the assumption that more inputs always equate to better knowledge.
2. Novel Framework (MARS): The authors design a method that introduces mono-source rewards as anchors to quantitatively measure information gain during advantage normalization. This enables adaptive regulation of different sources without requiring architectural redesign or off-policy correction.
3. Empirical Validation: Extensive experiments across diverse datasets (depth, infrared, multi-view, text-rich) and RL algorithms (GRPO, DAPO) demonstrate consistent performance improvements.

Experimental Results

The authors evaluated MARS on four multi-source tasks using the Qwen2.5-VL-3B and 7B models. Key findings include:

• Performance Gains: MARS achieved significant improvements over baseline RLVR methods. Specifically, it yielded 3.2% and 4.9% performance gains on average across GRPO and DAPO frameworks, respectively, on the multi-source reasoning metric.
• Specific Task Improvements: Notable gains were observed in infrared (3.8%) and multi-view (7.0%) datasets.
• Robustness: The method demonstrated superior robustness against visual degradations (Gaussian noise, motion blur, occlusion), maintaining high accuracy where baselines suffered performance drops.
• Efficiency: The approach requires only a single policy model and generates mono-source samples by modifying inputs, avoiding the storage overhead of experience replay or off-policy correction. The computational overhead is moderate, with the key benefit stemming from the anchor mechanism rather than increased trajectory volume.
• Qualitative Analysis: Case studies showed that MARS successfully prioritizes informative sources (e.g., infrared in low light) and suppresses irrelevant or noisy sources (e.g., white-out depth images), leading to more accurate reasoning and grounding compared to standard GRPO.

Significance and Claims

The paper claims that MARS offers a general and effective solution for improving visual reasoning performance in multi-source scenarios. By explicitly modeling the dynamic interaction between sources and quantifying information gain, the method resolves the conflict between "seeing more" and "knowing more."

The authors emphasize that their approach:

• Does not rely on specific training frameworks (compatible with GRPO and DAPO).
• Does not require architectural changes to the underlying model.
• Provides a principled way to handle modality conflicts and promotion, leading to more stable and interpretable optimization trajectories.
• Validates that dynamically leveraging mono-source rewards as anchors effectively guides models to focus on informative sources while suppressing noise, confirming that effective multi-source reasoning requires distinguishing between information gain and interference.