Technical Summary: What MLLMs Learn about When they Learn about Multimodal Reasoning

Problem Statement

Current evaluation of Multimodal Large Language Models (MLLMs) typically reduces performance to a single aggregate accuracy score. This approach implicitly treats reasoning as a unitary capability, obscuring the distinct sources of error: perception (extracting information from visual inputs), reasoning (operating on extracted information), and multimodal-specific interactions (coordination between modalities). Existing benchmarks rarely decompose these components with strict controls, often relying on varied input modalities without isolating specific skills. Consequently, it remains unclear how different training strategies (e.g., Supervised Fine-Tuning vs. Reinforcement Learning) influence specific multimodal reasoning capabilities.

Methodology: MATHLENS

To address this, the authors introduce MATHLENS, a controlled benchmark of textbook-style geometry problems designed to operationally decompose performance.

• Data Construction: MATHLENS is built upon the symbolic semantic state of FormalGeo-7K. Each problem instance $k$ is defined by a latent semantic state $S_k$ (a set of atomic facts, e.g., collinearity, angle measures) and a query operator $\phi_k$.
• Controlled Variants: From $S_k$, the benchmark generates four aligned annotations:
  1. Visual Diagrams ($C^{img}_k$): Rendered from geometric constraints to test perception.
  2. Textual Descriptions ($C^{txt}_k$): Definitively equivalent text encoding $S_k$, allowing for reasoning evaluation with trivialized perception.
  3. Multimodal Questions ($Q_k$): Requiring both modalities to solve.
  4. Perception Probes ($Q^{perc}_k$): Fine-grained questions targeting specific atomic facts in $S_k$ (e.g., "Which points are collinear?").
• Robustness Testing: The benchmark includes semantic diagram modifications (e.g., relabeling, rotation, adding auxiliary shapes) that preserve the ground-truth answer but alter visual form, testing semantic invariance beyond pixel-level augmentation.
• Error Decomposition: Model outputs are categorized into:
  • Perception Errors: Failure on perception probes.
  • Reasoning Errors: Success on probes but failure on text-only reasoning.
  • Multimodal-Specific (Integration) Errors: Success on both isolated perception and reasoning tasks, but failure on the combined multimodal task.
  • Residuals: A category capturing failures not explained by the measured perception and reasoning components.

The study evaluates 13 checkpoints across 7 model families (7–9B and 72B parameters), including variants trained with Direct RL, Multimodal SFT, Textual SFT, and combinations thereof.

Key Findings

The decomposition reveals that training strategies induce systematically different capability profiles that are invisible under aggregate accuracy:

1. Differential Impact of Training Strategies:

   • Reinforcement Learning (RL): Primarily improves perception and robustness to diagram variation. When paired with strong textual SFT, RL yields significant gains in perception. However, RL alone offers only modest improvements in both modalities.
   • Textual SFT: Despite lacking visual input during training, textual SFT indirectly strengthens perception. This is attributed to reflective reasoning, where enhanced cognitive strategies (e.g., self-correction) allow models to revisit and correct initial perceptual judgments.
   • Reasoning Gains: Reasoning improvements largely follow perceptual improvements under RL and do not exhibit distinct additional gains beyond these coupled improvements.

2. The Rise of Multimodal-Specific Residuals:
   As perception and reasoning capabilities improve, a growing fraction of remaining errors falls outside these measured components. These errors are categorized as multimodal-specific residuals (or integration failures). This suggests that apparent progress in multimodal reasoning often reflects shifting balances among components rather than uniform advancement.

3. Robustness Divergence:

   • RL enhances consistency under structural diagram variations.
   • Multimodal SFT reduces robustness, leading to overfitting on specific visual styles and a sharp drop in performance on modified diagrams.

4. Skill-Specific Gains:
   Perception gains are uneven. Training preferentially improves skills driven by direct geometric cues (e.g., collinearity, parallel lines). Tasks relying on symbolic marks or long-range visual associations (e.g., specific angle values, triangle detection) show limited gains unless combined with textual supervision.

Significance and Claims

The paper claims that MATHLENS provides a necessary framework for multi-axial evaluation of multimodal reasoning. By moving beyond scalar accuracy, the authors demonstrate that:

• Apparent progress in MLLMs is often a shift in the balance between perception, reasoning, and integration, rather than a uniform capability boost.
• Different training objectives (SFT vs. RL) shape these components in distinct ways, with RL being crucial for perceptual grounding and robustness, and textual SFT aiding perception through reflective reasoning.
• The persistence of multimodal-specific residuals indicates that current training paradigms struggle with cross-modal coordination, even when individual components improve.

The authors conclude that future work must address cross-modal coordination and perceptual capacity directly, motivated by the finding that standard aggregate metrics obscure these critical limitations. The paper does not claim to solve these issues but provides the diagnostic tools to identify them.