Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

Technical Summary: Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

1. Problem Definition

Spatio-temporal video grounding (STVG) requires a model to localize a natural-language query in both time (identifying the event interval) and space (tracking the referred target throughout that interval). While Multimodal Large Language Models (MLLMs) possess strong reasoning capabilities, applying them directly to long videos via frame-by-frame inference is computationally prohibitive and unstable.

The core challenges identified are:

• Computational Cost: Dense frame-level sampling creates an explosion of visual tokens, making long-video processing expensive.
• Instability: Direct generation of dense coordinates often leads to "drift," where the model loses track of the target identity or produces jittery trajectories due to small numerical errors in text-generated boxes.
• Supervision Mismatch: Standard supervised fine-tuning (SFT) optimizes token likelihood rather than task-specific metrics like temporal overlap or spatial trajectory consistency.

2. Methodology

The authors propose a practical pipeline that shifts the reasoning unit from the frame to the second, coupled with a three-stage training strategy and reinforcement learning (RL) verification.

A. Second-Level Formulation
Instead of processing $T$ frames, the video is converted into $K$ second-level units ($K = \lceil T/F \rceil$).

• Input: The model processes video units aggregated by second.
• Output: The model predicts a temporal span $[\hat{t}_s, \hat{t}_e]$ and a compact trajectory $\hat{B}$ consisting of one bounding box per second.
• Smoothing: A lightweight inter-second smoothing step is applied during inference to restore continuity and reduce jitter while preserving the predicted temporal boundaries.

B. Three-Stage Training Pipeline
The method utilizes a GLM-V-style architecture trained through three distinct stages:

1. Continual Pre-Training (CPT):

   • Goal: Build broad spatio-temporal grounding abilities.
   • Data: A mixture of STVG data, Multi-Object Tracking (MOT), Single-Object Tracking (SOT), and segmentation data.
   • Strategy: Videos are sampled at 1 FPS to match the second-level formulation. General video captioning and QA data are included to prevent the model from collapsing into a narrow coordinate-prediction task, preserving general conversational and reasoning abilities.

2. Supervised Fine-Tuning (SFT) with Corrected Reasoning:

   • Goal: Teach structured target selection and temporal reasoning.
   • Data: 20K examples from VidSTG and HC-STVG.
   • Mechanism: Advanced MLLMs (e.g., Gemini) generate Chain-of-Thought (CoT) traces focusing on what to track and when the event occurs.
   • Correction: Crucially, the generated spatial coordinates and temporal spans in the final output are replaced with ground-truth annotations. This separates useful semantic reasoning from unreliable generated geometry, ensuring the model learns the correct reasoning process without being penalized for geometric noise in the training data.

3. Reinforcement Learning (RL) with Verifier:

   • Goal: Directly optimize for grounding quality metrics.
   • Reward Function: A verifier scores predictions using a weighted sum of:
     • tIoU: Temporal Intersection over Union (overlap between predicted and ground-truth time windows).
     • mvIoU: Motion-aware spatial IoU (average IoU of boxes over the matched temporal intersection).
   • Optimization: The policy is optimized using a GRPO-style objective with asymmetric clipping. The combined reward ($tIoU + mvIoU$) provides a denser learning signal than using motion-aware rewards alone, helping the model first locate the correct event window before refining the trajectory.

C. Inference
The model outputs a structured answer containing the temporal span and second-level trajectory. Invalid states are discarded, and inter-second smoothing is applied. The output can remain at second-level granularity or be interpolated back to frame-level boxes if required by a benchmark.

3. Key Contributions

• Second-Level Tracking Formulation: A shift from frame-level to second-level units that significantly reduces sequence length and visual token costs while maintaining event structure and track continuity via smoothing.
• Corrected Reasoning Supervision: A training strategy that utilizes generated CoT traces for semantic reasoning but replaces generated coordinates with ground truth, preventing the model from learning from noisy geometric supervision.
• Verifier-Driven RL: An optimization stage using a verifier based on $tIoU + mvIoU$ to align the model's output directly with STVG evaluation metrics, rather than relying solely on token likelihood.
• Efficient High-Performance Model: Demonstration that a 9B parameter task-aligned model can outperform significantly larger general-purpose MLLMs (e.g., 397B Qwen3.5) on STVG tasks.

4. Experimental Results

The method was evaluated on VidSTG and HC-STVG benchmarks, as well as Video-MME-v2 to assess general video understanding.

• STVG Performance:

  • On VidSTG, the final RL model achieved an mtIoU of 31.35 and mvIoU of 23.79, outperforming all baselines including 397B and 1T parameter models.
  • On HC-STVG, the model achieved an mtIoU of 60.04 and mvIoU of 40.99, again surpassing larger generalist models.
  • The progression from SFT to RL showed that while SFT adapts the format, RL significantly improves metric alignment. The combined $tIoU + mvIoU$ reward converged faster and yielded better results than $mvIoU$ alone.

• General Video Understanding:

  • Evaluation on Video-MME-v2 showed the model improved over the GLM-4.1V-Thinking baseline in both non-linear reasoning (12.6 vs. 9.4) and simple accuracy (28.0 vs. 23.7), indicating the training pipeline did not overfit to localization at the expense of general reasoning.

• Ablation Studies:

  • FPS Sensitivity: Training at 1 FPS yielded the best results, balancing cost and temporal evidence.
  • Reward Design: The combination of temporal and motion-aware rewards was critical; using only motion-aware rewards resulted in weaker early learning signals.

5. Significance and Claims

The paper claims that the bottleneck in MLLM-based STVG is not merely model scale, but the formulation of the task and the design of the supervision signal. By decoupling semantic reasoning from geometric generation and optimizing directly for task metrics via RL, the authors demonstrate that a smaller, specialized model can achieve superior performance in long-video grounding compared to massive generalist models.

The authors position this work as a practical solution for the "missing piece" in MLLM-based STVG: handling long videos efficiently while producing usable, stable temporal and spatial grounding. They note that while the method is effective, future work could extend these reasoning capabilities to broader tasks like event parsing and interactive video agents, or adapt the approach for real-time and noisier open-world settings.