Technical Summary: TORM - Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models

Problem Statement

Video reasoning tasks require models to track motion, temporal order, and evolving visual states across frames. While Large Vision-Language Models (LVLMs) have succeeded in static image understanding, extending this to video remains challenging. Existing approaches often rely on externalizing reasoning through textual Chain-of-Thought (CoT), repeated frame reinsertion, temporal decomposition, or external tool use.

The paper identifies critical limitations in these external pipelines:

1. Inefficiency: They introduce substantial inference-time latency and engineering complexity.
2. Information Loss: They force rich, continuous temporal-visual evidence to be serialized into discrete text tokens or repeatedly re-encoded, potentially losing fine-grained dynamic information.
3. Trade-off: There is a fundamental tension between preserving rich intermediate visual-temporal information (beneficial for reasoning) and the efficiency of inference (hindered by generating or retrieving explicit evidence at test time).

Methodology: TORM

The authors propose TORM (Spatial-Temporal reasOning via inteRnalized Modeling), a two-stage framework designed to teach LVLMs to reason through bounded continuous latent trajectories rather than explicit textual CoT or external tools. The core intuition is that visual reasoning can occur implicitly within hidden states before verbalization.

Core Architecture

TORM augments the standard video-question-to-answer generation process with a short latent simulation segment containing $K$ latent tokens (set to $K=8$ in experiments).

• Inference: When the model generates a special <|latent_start|> token, it enters a "latent simulation mode." Instead of predicting discrete text tokens, the decoder generates continuous hidden states (latent tokens) which are fed back as input embeddings for subsequent steps. This continues until <|latent_end|> is generated, after which normal text decoding resumes for the final answer. This process is bounded to ensure fixed latency.
• No External Tools: Unlike other methods, TORM does not regenerate videos, re-insert frames, or call external tools during inference.

Data Pipeline

To train the model to internalize reasoning, the authors construct a training sequence with three aligned signals for each QA pair:

1. Keyframes: Question-relevant frames extracted from the source video.
2. Thought Video: A short, generated dynamic video visualizing the motion and state changes relevant to the question (generated using a teacher CoT plan and a video generation model like Wan 2.1).
3. Answer: The target answer token sequence.

Two-Stage Training Strategy

The training process is divided into two stages to first ground the latent space and then internalize the reasoning:

• Stage I: Latent-Grounded Supervised Training

  • The model is trained to align the predicted latent states ($z_i$) with the features of the generated thought video ($g_i$).
  • The thought video is encoded, and its features are pooled to match the fixed number of latent slots ($K$).
  • The loss function combines standard answer-token cross-entropy ($L_{ans}$) with an MSE loss ($L_{latent}$) to minimize the distance between latent states and thought-video features.
  • Goal: Ground the latent slots in dynamic visual evidence without requiring explicit textual CoT supervision.

• Stage II: Answer-Only Consolidation

  • The direct latent supervision (MSE loss) is removed.
  • The model is trained only on the answer-token loss ($L_{ans}$), while the latent slots remain in the input sequence.
  • Goal: Encourage the model to internalize the reasoning behavior learned in Stage I, making the latent process self-organized and robust for inference without explicit targets.

Key Contributions

1. TORM Framework: A video reasoning framework that performs internal dynamic simulation in continuous latent space, eliminating the need for explicit textual CoT or heavy tool-based inference.
2. Two-Stage Training Strategy: A novel approach combining latent alignment with thought-video supervision (Stage I) and answer-focused consolidation (Stage II) to internalize spatial-temporal reasoning.
3. Thought-Video Supervision Pipeline: A scalable method to align generated dynamic traces, keyframes, and QA annotations to train latent video reasoning.
4. Efficiency and Performance: Demonstrating that internalized reasoning can achieve high accuracy while significantly reducing inference overhead compared to tool-heavy or video-generation-based pipelines.

Experimental Results

The authors evaluated TORM on six benchmarks: VideoMME, MVBench, TempCompass, Video-Holmes, VideoEspresso, and MMVU.

• Performance: TORM consistently outperformed standard Supervised Fine-Tuned (SFT) baselines (e.g., Qwen2.5-VL-7B-SFT) across all benchmarks. It also matched or exceeded specialized models that rely on expensive test-time compute pipelines (e.g., LongVT, Video-Thinker).
  • Example: On VideoMME, TORM achieved 61.0% accuracy, surpassing the SFT baseline (55.4%) and competing with proprietary models like GPT-4o (71.9%) and Gemini-1.5-Pro (75.0%) while using an open-source backbone.
  • Example: On MMVU, TORM achieved 65.9%, outperforming Video-R1 (64.2%) and Qwen2.5-VL-CoT (60.0%).
• Latency: TORM demonstrated a massive reduction in inference latency.
  • LongVT and Video-Thinker required ~15–17 seconds per item.
  • TORM processed items at ~0.47 seconds per item (2.14 items/s), a ~30x speedup, by avoiding repeated frame retrieval and external tool calls.
• Ablation Studies:
  • Latent Size: A budget of 8 latent tokens provided the best balance. Larger budgets (16 or 256) degraded performance, suggesting that compact latent trajectories are more effective than retaining redundant visual tokens.
  • Training Stages: Both Stage I (grounding) and Stage II (consolidation) were necessary; using either stage alone resulted in lower performance.
  • Retrieval Analysis: Probes showed that the learned latent states retained video-specific temporal information better than textual representations, achieving a 71% Hit@1 in same-video retrieval compared to 29% for text.

Significance and Claims

The paper claims that TORM represents a practical step toward efficient native visual reasoning in video-language models. By shifting the burden of dynamic reasoning from inference-time tool orchestration to an internalized latent process, TORM addresses the trade-off between reasoning quality and computational efficiency.

The authors emphasize that:

• Dynamic visual evidence is used only during training to teach the latent states.
• At inference, the model reuses this learned behavior through a compact, fixed-budget latent trajectory without generating new videos or invoking external tools.
• This approach allows LVLMs to handle complex spatial-temporal reasoning tasks with the efficiency of a single-model decoding process, avoiding the latency and complexity of external reasoning pipelines.

The paper acknowledges limitations, noting that the quality of training depends on the generated thought videos and that current evaluation is focused on multiple-choice QA. Future work is suggested for open-ended reasoning and other architectures.