MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) have achieved significant success in standard video tasks like captioning and event recognition. However, they struggle with physics reasoning, particularly tasks requiring an understanding of:

• Motion Dynamics: How objects move, interact, and change state over time.
• Spatial Interactions: 3D spatial layouts and geometric relationships.
• Physical Plausibility: Distinguishing between real-world physics and artifacts in AI-generated content (AIGC).

Current VLMs often rely on superficial correlations between text and visual patterns rather than internalizing physical laws. They fail to generalize across diverse physical processes, especially when dealing with counterfactual scenarios (e.g., an apple floating upward) or AIGC videos containing physics violations. Existing datasets lack the dense, entity-level spatiotemporal annotations required to teach models these fine-grained dynamics.

2. Methodology: The MASS Framework

The authors propose MASS (Motion-Aware Spatial–temporal Grounding), a model-agnostic approach designed to inject structured physical cues into the VLM's language space without retraining the entire model from scratch.

A. Motion-Grounded Spatial-Temporal Representation

Instead of feeding raw video pixels alone, MASS extracts and serializes explicit physical signals:

1. Entity-Centric Visual Grounding: Uses Grounding-DINO to detect bounding boxes for queried entities and SAM2 to generate segmentation masks.
2. 3D Motion Tracking: Employs CoTracker3 to track point trajectories within grounded regions, calculating motion vectors and 3D displacements.
3. Depth Estimation: Uses Depth Anything V2 to estimate per-frame depth, projecting 2D motion into 3D space.
4. Serialization: These signals (3D positions, motion vectors, bounding boxes, and temporal segments) are converted into structured natural language templates. This allows the VLM to "read" the physics of the scene (e.g., "Entity X moved from <x1,y1,z1> to <x2,y2,z2> with vector <dx,dy,dz>") rather than inferring it from compressed visual tokens.

B. Training Pipeline

• Post-Training Strategy: The spatial-temporal encoders (tracking, depth) are kept frozen. The VLM backbone is fine-tuned to interpret these new textual cues.
• Reinforcement Fine-Tuning (RFT): The authors utilize Group Relative Policy Optimization (GRPO) with a temporal-aware reward function. This encourages the model to generate Chain-of-Thought (CoT) reasoning that aligns visual evidence with physical laws, rather than relying on language priors.
• Supervised Fine-Tuning (SFT): Used as a baseline comparison, but results show RFT is superior for complex physical reasoning.

3. Key Contributions

A. MASS-Bench (The Benchmark)

A comprehensive benchmark consisting of 4,350 videos (real-world and AIGC) and 8,361 free-form QA pairs.

• Data Sources: Combines real-world data (MotionSight, ActivityNet) and synthetic/AIGC data (VideoPhy2, VideoHallu, Sora, Wan2.2, COSMOS).
• Annotation Depth: Unlike previous benchmarks, MASS-Bench includes dense motion-grounding annotations:
  • Temporal segmentation.
  • Entity-level visual grounding (bounding boxes).
  • Full-video temporal profiles.
  • 3D motion attributes (start/end positions, displacement vectors).
• Categories: Questions are categorized into five levels of difficulty: Spatial Understanding, Temporal Understanding, Motion & Action Recognition, Physics Comprehension, and Physical Abnormality Detection.

B. The MASS Algorithm

A novel, plug-and-play module that bridges the gap between raw video dynamics and VLM reasoning by translating physical world cues into interpretable language representations.

C. Empirical Validation

Extensive experiments demonstrating that augmenting smaller models with MASS yields performance comparable to or exceeding much larger, closed-source models.

4. Experimental Results

The authors evaluated 12 state-of-the-art VLMs (ranging from 7B to 38B parameters, including GPT-4o and Gemini-2.5-Flash) on MASS-Bench.

• Performance Gains:
  • Qwen2.5-VL-7B + MASS achieved an overall accuracy of 61.93%, outperforming the previous best open-source model (Qwen3-VL-8B) by 8.7% and surpassing Gemini-2.5-Flash (63.18%) by a negligible margin (only ~1.2% difference), despite being a 7B parameter model.
  • LLaVA-OneVision-7B + MASS improved by 13.5% over its baseline.
• Ablation Studies:
  • RFT vs. SFT: Reinforcement Fine-Tuning (GRPO) significantly outperformed Supervised Fine-Tuning (SFT), proving that learning to reason about physics is more critical than just memorizing answers.
  • Modality: Combining visual tokens with motion-profile text yielded the best results; text-only or visual-only inputs were insufficient.
  • 3D vs. 2D: Using 3D motion representations (with depth) was crucial; 2D motion alone resulted in lower accuracy.
• Generalization: The method showed robustness against noise in motion cues and improved performance on out-of-distribution real-world benchmarks (MMVU, MVBench), though performance varied by model architecture.

5. Significance and Impact

• Bridging the Gap: MASS demonstrates that explicit, structured physical cues can compensate for the lack of "world model" intuition in standard VLMs.
• Efficiency: It offers a data-efficient path to physics reasoning without the prohibitive cost of pre-training massive models from scratch.
• AIGC Safety: The ability to detect physical abnormalities (e.g., in AIGC videos) is critical for verifying synthetic media and ensuring safety in AI-generated content.
• Future Direction: The work highlights that "reasoning" (via CoT and RFT) is the key mechanism for integrating visual evidence with physical commonsense, suggesting a new paradigm for training multimodal agents.

In summary, MASS redefines video understanding by treating physics not as an implicit feature to be learned from pixels, but as an explicit, structured signal that can be injected into the reasoning process of VLMs, achieving state-of-the-art results in physical comprehension and abnormality detection.