Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) excel at static visual understanding and spatial reasoning but struggle with "thinking in dynamics." While humans seamlessly perceive, track, and reason about objects in a physical 4D world (3D space + time), existing MLLMs face significant challenges when processing evolving scenes.

• Core Limitations: Models often fail to maintain consistent object representations across frames due to occlusion, ego-motion, and semantic ambiguity. They tend to produce fragmented understandings of motion, interactions, and temporal causality.
• Gap in Evaluation: Existing benchmarks primarily focus on scene-level temporal correlations or static spatial reasoning, lacking systematic evaluation of dynamic object grounding (tracking specific moving objects) and fine-grained spatio-temporal reasoning.
• Research Question: To what extent can MLLMs perceive, track, and reason about the physical 4D world coherently, and how can their performance be improved?

2. Methodology: Dyn-Bench

The authors introduce Dyn-Bench, a large-scale benchmark designed to rigorously evaluate MLLMs on dynamic understanding.

A. Dataset Construction

• Sources: Constructed from 8 diverse datasets (4 real-world 2D video segmentation datasets: DAVIS, SA-V, DynPose-100K, YouTube-VIS; and 4 synthetic 4D dynamic scene datasets: DynamicReplica, PointOdyssey, Spring, Total-Recon).
• Scale: Contains 1,000 high-quality videos, 7,000 Visual Question Answering (VQA) pairs, and 3,000 dynamic object grounding pairs.
• Filtering Pipeline: A multi-stage filtering strategy ensures data reliability:
  1. Geometric/Motion Stability: Checks for blur, frame rate consistency, and depth continuity.
  2. Dynamic Coverage: Ensures non-static cameras and evolving scenes with human/object interactions.
  3. VLM-Assisted Refinement: Uses VLMs to verify semantic coherence and motion validity.
  4. Human Validation: Final verification of video quality, mask consistency, and annotation accuracy.

B. Benchmark Structure (Three Levels)

Dyn-Bench evaluates models across three complementary dimensions, each integrating reasoning and grounding:

1. Dynamic Inter-Object Perception: Focuses on spatial relations and interactions between multiple moving objects (e.g., approaching, overtaking, occlusion).
2. Dynamic Object–Scene Tracking: Models how individual objects evolve within changing scenes (e.g., entering/leaving, functional transitions).
3. Dynamic Camera–Object Reasoning: Analyzes object behavior under varying camera motions (e.g., relative translation, rotation, depth changes).

C. Proposed Solutions for Improvement

To address the identified limitations, the paper proposes two structured integration approaches:

1. Spatio-Temporal Textual Cognitive Map (ST-TCM):
   • A framework that unifies spatial geometry, temporal semantics, and motion dynamics into a structured textual representation.
   • It reconstructs 3D object trajectories, camera poses, and inter-object relations from RGB-D inputs and converts them into JSON-like textual descriptions (e.g., "Object A moves forward-left at 3.8 m/s").
   • This serves as an auxiliary input to guide the MLLM's reasoning.
2. Mask-Guided Fusion:
   • A visual grounding strategy that overlays object segmentation masks onto raw video frames.
   • This explicitly directs the model's attention to moving entities, enhancing temporal continuity and motion perception.

3. Key Contributions

1. Dyn-Bench: The first benchmark to comprehensively evaluate MLLMs on dynamic object grounding and spatio-temporal reasoning simultaneously in realistic 4D environments. It covers three hierarchical levels of dynamic understanding.
2. High-Quality Dataset: A carefully curated dataset of 1k videos and 10k+ annotations derived from extensive 2D and 4D sources, featuring rigorous multi-stage filtering.
3. Diagnostic Analysis: A systematic evaluation revealing that current models (both general and region-level) struggle to balance reasoning and grounding, often producing inconsistent motion interpretations.
4. Novel Frameworks:
   • ST-TCM: Demonstrates that structured textual abstraction of geometric and motion cues significantly improves temporal coherence and causal reasoning.
   • Mask-Guided Fusion: Shows that integrating appearance and motion cues visually enhances fine-grained object tracking.

4. Experimental Results

The authors evaluated General, Spatial, and Region-level MLLMs (including GPT-4o, GPT-5, Qwen3-VL, UniPixel, Sa2VA, etc.).

• Performance Gaps:
  • General MLLMs: Strong in relational reasoning but weak in precise dynamic grounding.
  • Spatial MLLMs: Better at geometry-dependent tasks but fail in camera-object interaction and motion-centric reasoning.
  • Region-level MLLMs: Achieve the best grounding accuracy but still struggle with complex spatio-temporal reasoning without auxiliary guidance.
• Impact of Proposed Methods:
  • ST-TCM: Significantly boosts performance. For example, adding motion and spatial cues (M+S) to Qwen3-VL-8B improved reasoning accuracy, while the full ST-TCM (T+M+S) yielded the highest gains.
  • Mask-Guided Fusion: Improved all task categories, with the largest gains in Inter-Object and Camera-Object reasoning (up to ~3-4% absolute improvement over raw video baselines).
• Error Analysis: Models primarily fail due to temporal reasoning errors (treating sequences as discrete frames), spatial grounding errors (inaccurate distance estimation), and relational errors (missing causal dependencies).

5. Significance and Future Directions

• Bridging the Gap: Dyn-Bench shifts the focus from static scene understanding to the dynamic 4D world, providing a necessary standard for evaluating "thinking in dynamics."
• Methodological Insight: The paper proves that structured integration (combining explicit geometric/motion cues via text or masks) is more effective than conventional prompting strategies (like Chain-of-Thought) for dynamic tasks.
• Future Work: The findings suggest that future MLLMs must tightly integrate dynamic-object perception with temporal reasoning, moving toward unified architectures that jointly model motion dynamics, relational structures, and higher-level temporal cognition for physically grounded reasoning.

In summary, this work establishes that while current MLLMs have strong static capabilities, they lack the structured mechanisms to "think in dynamics." Dyn-Bench and the proposed ST-TCM/Mask-Guided approaches provide a roadmap for building models capable of coherent, physically grounded reasoning in evolving 4D environments.