STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering

1. Problem Statement

The paper addresses a critical bottleneck in applying Group Relative Policy Optimization (GRPO) to Large Multimodal Models (LMMs) for Video Question Answering (VQA).

• The Advantage Collapse Problem: Standard GRPO improves model performance by sampling multiple textual responses for a fixed input and computing relative advantages based on reward variance within that group. However, in video reasoning tasks, generated responses often exhibit low variance (e.g., all are correct or all are incorrect). When the reward variance approaches zero, the advantage estimates either explode (due to division by a near-zero standard deviation) or collapse to zero, leading to unstable or stalled training.
• Limitation of Text-Only Diversity: Existing methods attempt to solve this by increasing textual diversity or modifying reward shaping. However, they treat the video input as a static, fixed entity. This ignores the fundamental nature of video, which possesses rich spatiotemporal structure. By only varying the text, the model is not exposed to diverse visual evidence within a single optimization step, limiting its ability to learn robust reasoning anchored in visual data.

2. Methodology: The STRIVE Framework

STRIVE (SpatioTemporal Reinforcement with Importance-aware Variant Exploration) proposes a paradigm shift: moving diversity from the output space (text) to the input space (video).

A. Dual-Grouping Paradigm

Instead of generating $G$ textual responses for a single video, STRIVE constructs a $G \times M$ matrix of responses:

1. Variant Construction: For a given video $v$ and question $q$, the system generates $M$ distinct spatiotemporal variants of the video ($\tilde{v}_1, \dots, \tilde{v}_M$).
2. Joint Generation: The policy model generates $G$ textual responses for each of the $M$ video variants.
3. Joint Normalization: The advantage is computed over the entire pool of $G \times M$ responses, rather than just the $G$ responses for a single video.
   • Formula: $A_{i,j} = \frac{r_{i,j} - \text{mean}(\{r_{i,j}\})}{\text{std}(\{r_{i,j}\})}$
   • Benefit: This ensures a consistently positive reward variance ($\text{Var}(r) > 0$) even if textual answers are uniform, because the visual inputs differ. It acts as an implicit consistency regularizer, forcing the model to generate answers that remain correct across different valid visual perspectives.

B. Importance-Aware Variant Construction

To ensure the visual variants are semantically meaningful and not just random noise, STRIVE employs a temperature-scaled, importance-aware sampling strategy:

1. Relevance Scoring: A scoring function $S(v_f, q)$ computes a relevance score for each frame $f$ based on cross-modal embedding similarity between the frame and the question.
2. Segmented Sampling: The video is divided into $K$ temporal segments.
3. Probabilistic Selection: Instead of deterministically picking the highest-scoring frame (which reduces diversity), STRIVE samples a representative frame from each segment using a softmax distribution scaled by temperature $\tau$.
   • Low $\tau$: Exploits the most relevant frames.
   • High $\tau$: Encourages broader temporal exploration.
     This balances semantic focus (attending to question-critical events) with temporal coverage (preventing overfitting to isolated moments).

C. Transformation Strategies

The paper explores three types of transformations to create variants:

1. Deterministic Temporal: Strided sampling with staggered offsets.
2. Stochastic: Random temporal cropping combined with spatial augmentations (color jitter, affine transforms).
3. Importance-Based: The proposed method using relevance scores to guide sampling.

3. Key Contributions

1. Structured Spatiotemporal Exploration: The first framework to integrate visual input perturbations directly into the group-based RL optimization loop for VQA, solving the "zero-advantage" bottleneck without relying solely on textual diversity.
2. Implicit Consistency Regularization: By normalizing across visual variants, the method enforces that the model's reasoning must be robust to spatiotemporal changes, preventing reliance on spurious language priors.
3. Importance-Aware Sampling: A novel mechanism that guides visual exploration toward question-relevant frames while maintaining necessary stochasticity for stable gradient estimation.
4. Plug-and-Play Compatibility: The framework is designed to work with existing policy optimization algorithms (like GRPO and Dr.GRPO) as a variant construction module.

4. Experimental Results

The authors evaluated STRIVE on six challenging benchmarks: VideoMME, TempCompass, VideoMMMU, MMVU, VSI-Bench, and PerceptionTest.

• Base Models: Qwen2.5-VL-7B and Qwen3-VL-8B.
• Performance Gains:
  • Qwen2.5-VL-7B: STRIVE achieved an average score of 59.8, outperforming the standard GRPO baseline (59.0) and the supervised fine-tuned model (57.6).
  • Qwen3-VL-8B: STRIVE achieved 65.2, significantly outperforming the GRPO baseline (64.3), which actually showed performance drops on specific benchmarks compared to the base model.
• Stability Analysis:
  • Reward Variance: STRIVE maintained consistently higher reward standard deviation throughout training compared to GRPO, which saw variance collapse as training progressed.
  • Zero-Advantage Steps: STRIVE drastically reduced the fraction of training steps with zero advantages (gradient starvation), whereas GRPO frequently encountered this issue.
  • Gradient Norm: STRIVE produced smoother, more stable gradient trajectories, avoiding the chaotic spikes seen in standard GRPO.
• Ablation Studies:
  • Importance-Based vs. Stochastic: Importance-based grouping (using cross-modal similarity) outperformed pure stochastic grouping, proving that guided exploration is superior to blind randomization.
  • M vs. G Trade-off: A balanced configuration ($M=2$ visual variants, $G=4$ text generations) yielded the best results. Too many visual variants ($M=4, G=2$) reduced the reliability of the advantage baseline, while too few ($M=1$) failed to prevent advantage collapse.

5. Significance

STRIVE represents a principled advancement in multimodal reinforcement learning. It demonstrates that for video reasoning, visual diversity is as critical as textual diversity. By restructuring the optimization space to include spatiotemporal variants, STRIVE provides a stable learning signal where traditional methods fail. This approach not only improves performance on complex reasoning tasks but also offers a generalizable mechanism to stabilize RL training in any domain where input structure can be systematically varied to enrich reward signals. The paper suggests that future LMM alignment should explicitly leverage the structural properties of the input modality (e.g., time in video, 3D structure in scenes) rather than treating inputs as static tokens.