Technical Summary: Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training

1. Problem Statement

Post-training techniques, specifically Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), have significantly enhanced the reasoning capabilities of frontier Vision-Language Models (VLMs). However, these gains have not translated proportionally to perception abilities. Recent studies indicate that post-training may yield limited improvements or even degrade perception performance, creating a bottleneck for end-to-end visual reasoning. The central question addressed by this work is: Why does post-training disproportionately favor reasoning over perception, and how can this asymmetry be mitigated?

The authors argue that perception errors severely limit the effectiveness of advanced reasoning. To investigate this, they introduce a controlled diagnostic framework that disentangles perception from reasoning, allowing for the independent evaluation and optimization of each component.

2. Methodology

2.1 Controlled Diagnostic Framework

The authors propose a framework designed to separate perception ($p$) and reasoning ($r$) within the model's output.

• Disentangled Outputs: The output sequence is structured as $y = [p, r]$, where $p$ is a textual transcription of visual information and $r$ is the reasoning process, separated by special tokens. This enforces a "perception-before-reasoning" order.
• Evaluation Metrics:
  • Perception Accuracy ($a_p$): Measured by comparing the model's generated perception $p$ against a ground-truth canonical representation $p^*$.
  • Counterfactual Reasoning Accuracy ($a_r$): To isolate reasoning ability from perception errors, the model is conditioned on the ground-truth perception $p^*$ to generate reasoning $r'$. The accuracy is then measured as $Acc(r' | p^*)$.
  • End-to-End Accuracy ($a$): The standard accuracy of the full pipeline.
• Synthetic Tasks: To ensure ground-truth annotations and programmatic evaluation, the authors utilize two synthetic tasks:
  1. Graph Coloring (GC): Input images of graphs; $p^*$ is the edge list; correctness requires valid node coloring.
  2. Sudoku: Input images of grids; $p^*$ is the partially filled grid; correctness requires a valid solution.
     These tasks are rendered programmatically from text-based problems to maintain a canonical $p^*$.

2.2 Training Paradigms

The study evaluates two dominant post-training methods:

• Supervised Fine-Tuning (SFT): Optimizes token-averaged cross-entropy loss over the output sequence.
• Reinforcement Learning (RL): Uses Group Relative Policy Optimization (GRPO) with end-to-end accuracy as the primary reward signal.

2.3 Hypothesized Mechanisms and Interventions

The authors investigate two specific mechanisms driving the asymmetry:

1. Hypothesis 1: Token Imbalance (SFT)

   • Mechanism: In Chain-of-Thought (CoT) supervision, perception tokens occupy a significantly smaller fraction of the total output compared to reasoning tokens. Since SFT uses token-averaged loss, perception receives a weaker optimization signal.
   • Intervention: Loss Reweighting. The SFT loss is modified to $L_{SFT, \lambda} = \lambda \frac{L_p}{|p|} + (1-\lambda) \frac{L_r}{|r|}$, where $\lambda$ increases the weight of perception tokens. They also test NGDiff, a dynamic multi-task balancing method based on gradient norms.

2. Hypothesis 2: Reward Coupling (RL)

   • Mechanism: The outcome reward (end-to-end accuracy) correlates more strongly with reasoning correctness than with perception correctness. Perception only influences the reward if the subsequent reasoning is also correct, leading to noisy or misaligned credit assignment for perception.
   • Intervention: Reward Augmentation. The reward function is modified to $R_\alpha(y|x) = \alpha \cdot a_p + (1-\alpha) \cdot a$, explicitly adding a perception signal. They also evaluate surrogate rewards (CLIP similarity, self-rewarding, teacher-based rewards) for scenarios where ground-truth perception rewards are unavailable.

3. Experimental Results

3.1 Empirical Setup

Experiments were conducted on Qwen3-VL-2B and InternVL3.5-2B using the GC and Sudoku tasks. Datasets included 2,000 SFT training instances and 16,000 RL training instances per task.

3.2 Findings on SFT

• Asymmetry Confirmation: Standard SFT substantially improves reasoning accuracy while yielding minimal gains in perception accuracy.
• Token Imbalance Validation: Analysis revealed perception accounts for only ~2.5% of tokens in CoT traces, contributing merely 1.3% of the total loss and 8.5% of the gradient norm.
• Intervention Efficacy:
  • Increasing the perception weight ($\lambda$) improves perception accuracy but creates a trade-off where excessive weighting harms reasoning.
  • Optimal Balance: End-to-end accuracy peaks at a moderate upweighting of perception.
  • Results: Loss reweighting improved end-to-end accuracy by 10.0–13.8 points over standard SFT. The dynamic NGDiff method further improved this, achieving gains of up to 18.2 points without manual hyperparameter tuning.

3.3 Findings on RL

• Asymmetry Confirmation: RL also improves reasoning more than perception.
• Token Imbalance Rejection: Unlike SFT, reweighting token probabilities in the RL objective did not mitigate the asymmetry and often destabilized training. This indicates token count is not the primary driver in RL.
• Reward Coupling Validation: Pearson correlation analysis showed the outcome reward correlates strongly with reasoning ($r \approx 0.65\text{--}1.00$) but weakly with perception ($r \approx 0.34\text{--}0.43$).
• Intervention Efficacy:
  • Reward Augmentation: Adding a perception reward ($\alpha > 0$) improved perception accuracy. Similar to SFT, a trade-off exists, but end-to-end performance peaks at a non-zero $\alpha$.
  • Results: Reward augmentation improved end-to-end accuracy by up to 6.0 points over standard RL.
  • Surrogate Rewards: When ground-truth perception rewards were unavailable, teacher-based rewards (using a stronger model to generate $p^*$) provided the most reliable signal, yielding 3.2 points of improvement. CLIP-based similarity provided negligible signal.

4. Key Contributions

1. Diagnostic Framework: Introduced a controlled framework using synthetic tasks to disentangle and independently evaluate perception and reasoning in VLM post-training.
2. Mechanism Identification: Identified consistent perception-reasoning asymmetry across SFT and RL, tracing the causes to distinct mechanisms: token imbalance in SFT and reward coupling in RL.
3. Targeted Interventions: Demonstrated that specific, paradigm-aware interventions can rebalance these components:
   • SFT: Loss reweighting (static or dynamic via NGDiff).
   • RL: Perception-aware reward augmentation (using ground-truth or reliable surrogates).
   • These interventions improved end-to-end visual reasoning performance by up to 18.2 points (SFT) and 6.0 points (RL).

5. Significance and Claims

The paper claims that the observed asymmetry is a fundamental consequence of default post-training objectives rather than a model-specific artifact. The authors argue that relying solely on standard SFT or RL objectives is insufficient for robust end-to-end visual reasoning because perception remains the dominant bottleneck.

The significance of the work lies in providing concrete, actionable strategies to mitigate this imbalance. By diagnosing the specific failure modes (token scarcity in SFT, weak reward signals in RL), the authors suggest that future VLM post-training pipelines must incorporate explicit balancing mechanisms—such as loss reweighting or perception-aware rewards—to unlock the full potential of visual reasoning. The study emphasizes that these interventions are practical and can be integrated into real-world pipelines whenever perception supervision or proxy signals are available.

Limitations: The authors note that the study relies on synthetic tasks (Graph Coloring and Sudoku) to isolate variables. While this abstraction limits direct benchmarking on real-world diversity, it is necessary to establish causal mechanisms that are obscured in complex, ambiguous real-world data. The proposed mechanisms (token imbalance and reward coupling) are argued to be general properties of post-training objectives, suggesting the interventions are applicable beyond the synthetic setting.