Technical Summary: Annotations as Rollouts (OraRL)

Problem Statement

Unified video perception requires Multimodal Large Language Models (MLLMs) to perform fine-grained tasks such as temporal localization, spatial grounding, object tracking, and segmentation. While recent generalist MLLMs integrate these capabilities, they often underperform task-specific specialists, suggesting that model scale alone is insufficient and that post-training alignment is the critical bottleneck.

Current post-training paradigms face two primary limitations:

1. Supervised Fine-Tuning (SFT): SFT treats annotations as maximum-likelihood targets, enforcing output formats but failing to distinguish between near-correct and clearly incorrect predictions. It lacks task-level supervision to guide the model toward precise intervals or masks.
2. Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO): Existing video RL methods (e.g., GRPO) sample multiple on-policy rollouts per prompt and compare their rewards. However, these methods rely solely on the quality of the sampled on-policy group. Since on-policy rollouts rarely recover the precise intervals, boxes, or masks specified by ground-truth (GT) annotations, many training groups lack a reliable positive anchor.
3. Chain-of-Thought (CoT) Limitations: While CoT generation is used to improve reasoning, it lengthens rollouts, increasing training and inference costs without guaranteeing performance gains in fine-grained perception tasks.

A direct attempt to incorporate GT annotations into the RL rollout group as "oracle" rollouts fails due to advantage inversion. When a high-reward oracle is included in the group normalization, it raises the baseline reward. Consequently, on-policy rollouts that are better than the original policy (but worse than the oracle) receive negative advantages, suppressing useful exploration and collapsing learning toward simple imitation.

Methodology: OraRL

The paper introduces OraRL (Annotation-as-Rollout Reinforcement Learning), a paradigm that treats every annotation as a reliable positive target (an "oracle rollout") while preventing the baseline shift that causes advantage inversion.

Core Mechanisms

1. Annotation-as-Rollout Construction:
   Instead of using annotations merely as scoring references, OraRL serializes each annotation $y$ into the model's response format to create an oracle rollout $o_{gt}$. This oracle is appended to the group of $n$ on-policy rollouts, creating an augmented group of size $n+1$. This provides a reliable positive target for every query without reducing on-policy exploration.

2. Decoupled Advantage Estimation:
   To solve the advantage inversion problem, OraRL decouples the advantage calculation:

   • Policy Baseline: The baseline $\mu_{op}$ and standard deviation are computed only from the on-policy rewards, excluding the oracle. This ensures that any on-policy rollout outperforming the current policy receives a non-negative advantage.
   • Directional Gain: The gap between the oracle reward and the on-policy distribution is encoded as a directional gain $g_q$. This gain scales the advantages of on-policy rollouts that are above the on-policy mean, amplifying the signal when the oracle is significantly better than the current policy.
   • Detached Oracle Advantage: A separate, bounded advantage term $A_{gt}$ is computed for the oracle rollout itself. Its scale is calibrated by a weight $w_q$ (based on the remaining gap between policy and oracle) and capped by the strongest on-policy signal to prevent the oracle from dominating the update.

3. Sign-Balanced Pruning:
   To improve efficiency, OraRL prunes the rollout group before back-propagation. Unlike magnitude-only pruning (which might retain only positive or only negative samples), OraRL employs sign-balanced pruning:

   • The oracle is always retained.
   • The remaining slots are filled by retaining the strongest positive and negative on-policy rollouts, ensuring the gradient update preserves both reinforcing and suppressive signals.
   • A post-selection moment correction is applied to re-center the advantages and rescale them to match the pre-pruning statistics, preventing bias in the update magnitude.

4. Efficiency:
   By pruning the group (e.g., retaining 4 out of 9 rollouts) and avoiding CoT generation, OraRL achieves a step time of only $2.2\times$ that of SFT, compared to $4.9\times$ for GRPO with CoT.

Key Contributions

1. Annotation-as-Rollout: A task-independent mechanism that converts annotations into oracle rollouts, providing reliable positive supervision without requiring CoT or sacrificing on-policy exploration.
2. Advantage Inversion Analysis: Identification of the "advantage inversion" phenomenon in naive oracle mixing and a solution via decoupled advantage estimation that separates policy advantages from oracle guidance.
3. Sign-Balanced Pruning: A pruning strategy that retains both advantage signs and the oracle, coupled with moment correction, yielding a $1.48\times$ speedup over full-group optimization.
4. Video-ORA: A unified video perception model trained with OraRL, demonstrating consistent improvements across model scales (0.8B to 9B) and data budgets.

Experimental Results

The authors trained Video-ORA (based on Qwen3.5 backbones) and evaluated it across seven task families: temporal grounding, spatial grounding, segmentation, visual tracking, spatial-temporal grounding, video QA, and spatial intelligence.

• Performance Gains:

  • Temporal Grounding: Video-ORA-9B achieved an mIoU of 61.8, 63.6, and 72.5 on the three TimeLens benchmarks, surpassing the temporal specialist TimeLens2-8B and proprietary models like GPT-5 and Gemini-3-Pro.
  • Visual Tracking: On GOT-10k, Video-ORA-9B achieved an Average Overlap (AO) of 78.2, outperforming the previous best open-source model (OneThinker-8B) by 5.2 points.
  • Segmentation: It achieved leading scores on RefCOCO (cIoU 79.4) and MeViS (J&F 61.3).
  • Spatial Intelligence: On VSI-Bench, Video-ORA-9B scored 73.1, outperforming GPT-5 (55.0) and Gemini-3-Pro (55.1).
  • Video QA: It ranked first among open-source models on five of seven Video QA benchmarks.

• Scaling and Efficiency:

  • Model Scaling: Video-ORA improved over its backbone at all scales (0.8B, 2B, 4B, 9B), with macro-average gains increasing with model size.
  • Data Scaling: OraRL outperformed GRPO and SFT across data budgets up to 100k prompts.
  • Inference Latency: Without CoT, Video-ORA-9B reduced median end-to-end latency on 10-minute videos from 29.0s (CoT backbone) to 24.3s.

Significance and Claims

The paper claims that OraRL establishes "annotation-as-rollout" as an efficient and scalable reinforcement learning principle for unified video perception. The authors argue that:

• Fine-grained precision in video perception is determined primarily by task-aligned post-training rather than model scale alone.
• Direct oracle integration, when handled correctly (via decoupled advantages), provides a reliable positive target that overcomes the scarcity of high-quality on-policy rollouts.
• CoT reasoning is not necessary for these tasks; direct oracle supervision yields better accuracy and efficiency.
• The method is generalizable across different backbone families (Qwen3-VL, Qwen3.5) and task types (localization, tracking, QA, spatial reasoning).

The authors note limitations, specifically that the current formulation assumes annotations can be serialized as valid oracle rollouts and evaluated by scalar rewards, and that behavior under ambiguous or noisy supervision has not been evaluated. They also acknowledge that while Video-ORA leads in many open-source comparisons, some complex spatial reasoning tasks still lag behind the strongest proprietary models.