Technical Summary: CARE (Competence-Aware Reward Shaping)

1. Problem Statement

In multimodal video reasoning, particularly within Video Question Answering (VideoQA), models must process heterogeneous inputs, track temporal changes, and compose intermediate inferences. While recent post-training methods utilize Reinforcement Learning (RL) to encourage explicit Chain-of-Thought (CoT) reasoning, existing approaches rely on static reasoning-length control strategies (e.g., fixed rewards, penalties, or thresholds).

The paper identifies a critical competence–constraint mismatch in these static designs:

• Early Training: When model competence is low, static brevity penalties may prematurely truncate useful exploration trajectories, preventing the model from discovering necessary inference paths.
• Late Training: As the model becomes more competent, static incentives for long reasoning encourage redundant narration, self-verification padding, and inefficient decoding, consuming tokens without improving solution quality.
• Static Normalization: Fixed length normalization fails to account for instance difficulty, often penalizing necessary long reasoning on hard samples or failing to penalize verbosity on easy ones.

The core problem is that the desirable "reasoning budget" is non-stationary; it should expand during exploration and contract during consolidation, yet current RL frameworks treat it as a fixed hyperparameter.

2. Methodology: CARE Framework

The authors propose CARE (Competence-Aware Reward Shaping), a training-time reward framework integrated into the GRPO (Group Relative Policy Optimization) pipeline. CARE dynamically adjusts the reward preference for reasoning length based on the model's evolving competence, without requiring inference-time controllers.

Core Components

1. Competence Monitor (EMA):

   • Instead of using instantaneous batch accuracy (which is noisy), CARE maintains a smoothed estimate of model competence ($P_{b,t}$) using an Exponential Moving Average (EMA) of group pass rates.
   • This provides a stable global state variable to track long-term capability trends.

2. Stage-Aware Routing:

   • Training is divided into four progressive stages based on competence thresholds: Novice, Explorer, Proficient, and Master.
   • Novice/Explorer: The reward strongly favors longer reasoning to maximize the exploration space.
   • Proficient/Master: The reward shifts to penalize redundancy, favoring concise, information-dense reasoning.
   • This routing ensures the model is not compressed before it has acquired sufficient reasoning patterns.

3. Difficulty-Aware Effort Scoring:

   • Raw token length is normalized relative to the batch context to avoid conflating verbosity with intrinsic task complexity.
   • The normalized effort score ($z$) is calculated using batch statistics (average length of correct responses, 75th percentile) and a stabilizing floor ($T_{floor}$) to prevent premature truncation.
   • A sigmoid function modulates the reward based on the current stage's preference coefficient ($k_s$).

4. Posterior Amplifier:

   • To distinguish routine outcomes from unexpectedly strong performance, CARE introduces an amplification factor ($A$).
   • If a group's pass rate significantly exceeds the historical competence estimate ($\Delta_{diff}$), the reward is amplified. This is particularly useful for reinforcing successful exploration on historically difficult samples.

5. Final Reward Assembly:

   • The total reward combines task accuracy ($R_{acc}$), formatting rewards, and the CARE effort reward ($R_{CARE}$).
   • The effort term is gated by correctness: $R_{CARE} = R_{raw} \cdot R_{acc}$.
   • The final objective is $R_{total} = R_{acc} + R_{format} + R_{CARE} + \lambda_{temp}R_{temporal}$.

3. Key Contributions

1. Identification of Mismatch: The paper formally identifies that static reasoning-length rewards impose a fixed budget preference that conflicts with the non-stationary nature of model competence in RL.
2. CARE Framework: A novel reward shaping mechanism that transitions from exploration-oriented long-form reasoning to efficiency-oriented concise reasoning via competence-based stage routing, difficulty normalization, and posterior amplification.
3. Emergent Adaptivity: The method induces an adaptive reasoning policy where the model self-organizes to allocate more reasoning to hard instances and less to easy ones, achieving this without explicit inference-time controllers.

4. Experimental Results

Experiments were conducted on the Qwen2.5-VL-7B backbone using the Video-R1 dataset (260k pairs) and evaluated across multiple benchmarks.

Performance Gains

• Video Reasoning: CARE outperformed the Video-R1 baseline significantly on reasoning-intensive datasets:
  • VSI-Bench: Improved from 30.3 to 33.9.
  • VideoMMMU: Improved from 47.2 to 50.2.
  • MMVU: Improved from 63.5 to 64.2.
• General Video Understanding: Consistent improvements were observed on MVBench, TempCompass, and VideoMME, demonstrating that efficiency gains do not come at the cost of general perception.
• Scaling: Performance gains were consistent when increasing input frames from 16 to 64, indicating robustness to richer temporal contexts.

Efficiency and Training Dynamics

• Token Efficiency: CARE achieved a significant reduction in token consumption (up to ~41% on some benchmarks) while maintaining or improving accuracy.
• Inverted-U Trajectory: Training dynamics revealed a characteristic "expansion-compression" cycle. Reasoning length initially increased (exploration) and then decreased (consolidation) as competence grew, contrasting with the static baseline which maintained a fixed, often suboptimal, length.
• Ablation Studies:
  • Removing the competence-aware routing (Fixed-step) caused significant performance drops, confirming that static stage transitions are insufficient.
  • Removing the Posterior Amplifier or EMA smoothing led to instability or reduced performance on hard instances.
  • The Stabilizer Floor ($T_{floor}$) was critical to prevent training collapse into trivial, short responses.

5. Significance and Claims

The paper claims that CARE provides a principled balance between reasoning exploration and execution efficiency. Its primary significance lies in:

• Adaptive Budget Allocation: Demonstrating that reasoning length should be treated as a controllable budget that evolves with the policy, rather than a static regularization term.
• Training-Time Internalization: Showing that adaptive behavior can be "internalized" into the policy during training, eliminating the need for complex inference-time controllers or post-hoc truncation.
• Robustness: The framework stabilizes RL training, preventing the oscillation between over-thinking and under-thinking, and yields shorter yet more informative reasoning traces at convergence.

The authors modestly note that while the current hyperparameter settings provide a strong balance, the absolute optimum may vary with model scale or data mixtures. The core contribution is the validation of competence-aware automatic reward shaping as an interpretable and effective training mechanism for multimodal reasoning systems.