💻 computer science

4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

4DThinker is a novel framework that enhances vision-language models' dynamic spatial reasoning by enabling them to "think with 4D" through internally simulated latent mental imagery, supported by an annotation-free data pipeline, Dynamic-Imagery Fine-Tuning, and 4D Reinforcement Learning.

Original authors: Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xiang An, Bo Li, Xin Xie, ZiDong Wang, Mingze Sun, Shuang Chen, Hongyu Li, Xiaobin Hu, Ruqi Huang

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xiang An, Bo Li, Xin Xie, ZiDong Wang, Mingze Sun, Shuang Chen, Hongyu Li, Xiaobin Hu, Ruqi Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a video of a busy street. A car drives past a red building. If you ask a standard AI, "Is the car getting closer to the building, or is the camera moving away?" the AI might get confused. It usually tries to answer by writing a long, wordy paragraph describing the scene. But words are clumsy tools for describing complex movement in 3D space; they are like trying to describe a dance by only writing down the steps, rather than actually seeing the dancer move.

4DThinker is a new way of teaching AI to "think" about moving videos. Instead of just writing words, the AI learns to create a mental movie inside its own brain to solve the problem.

Here is how it works, broken down into simple steps:

1. The Problem: Words vs. Motion

Current AI models are great at reading and writing, but they struggle with dynamic spatial reasoning. This is the ability to understand how objects and the camera move relative to each other over time.

The Old Way: The AI tries to describe the motion entirely in text. It's like trying to explain a rollercoaster ride by only listing the words "up," "down," and "fast." It's often imprecise and confusing.
The Other Old Way: Some researchers attach extra "glasses" (external geometric tools) to the AI to help it see 3D shapes. But this makes the AI slow and clunky, like giving a runner a heavy backpack.

2. The Solution: "Thinking with 4D"

4DThinker teaches the AI to simulate motion internally, using what the authors call "Dynamic Mental Imagery."

The Analogy: Imagine you are watching a car drive by. Instead of just saying "the car moved left," you close your eyes and visualize the car's path in your mind. You see the car getting smaller as it drives away. 4DThinker does exactly this, but inside its computer code. It creates a hidden, invisible "movie" that it runs through its brain to figure out the answer.

3. How They Taught It (The Three Steps)

Step A: Making the Practice Data (No Humans Needed)
To teach the AI, they needed thousands of videos with questions and answers. Usually, humans have to label these videos, which is slow and expensive.

The Trick: They built an automated pipeline that takes raw videos and uses other AI tools to automatically find moving objects (like a person on a bike) and stationary objects (like a building). It then creates "highlighted" versions of the video frames to show the AI exactly what to focus on. This created a massive library of practice problems without a single human drawing a line on a screen.

Step B: The "Warm-Up" (DIFT)
First, they taught the AI to link its words with its mental movies.

The Analogy: Think of this like a student learning to draw while listening to a teacher. The AI is shown a video frame, and it has to generate a "mental image" (a hidden code representing the visual) and a text answer at the same time. It learns that to answer correctly, it must "see" the movement in its mind before it speaks.

Step C: The "Coach" (4D Reinforcement Learning)
Once the AI knows the basics, they let it practice on harder, more complex videos where both the camera and the objects are moving.

The Trick: They didn't give the AI the answers. Instead, they acted like a coach. If the AI got the answer right, it got a "reward." If it got it wrong, it got no reward. The AI figured out on its own how to use its mental movies to get the reward. Crucially, they only let the AI change its words during this practice, keeping the "mental movie" part stable so it didn't get confused.

4. The Results

The paper tested this new AI on several difficult video quizzes about movement, distance, and direction.

The Outcome: 4DThinker consistently beat other top AI models, including very expensive "proprietary" ones.
Why it won: It didn't need extra tools or heavy backpacks. It simply got better at "imagining" the scene in 4D (3D space + time) inside its own brain, just like a human does when they try to figure out if they are moving or if the world around them is moving.

Summary

4DThinker is a framework that lets AI models stop trying to describe motion with clumsy words and start simulating motion in their own minds. By training the AI to generate "mental movies" alongside its answers, it becomes much better at understanding how the physical world moves and changes.

Technical Summary: 4DThinker

Problem Statement

Dynamic spatial reasoning from monocular video—understanding how cameras and objects move and interact in 3D space over time—is a critical capability for bridging visual intelligence with the physical world. Despite advances in Vision-Language Models (VLMs), current systems struggle with basic dynamic reasoning. Existing approaches generally fall into two categories, both with significant limitations:

Text-Only Reasoning: Methods that verbalize spatial-temporal reasoning entirely as text (e.g., Chain-of-Thought) are inherently verbose and imprecise for conveying complex dynamics like 3D trajectories and camera ego-motion.
External Geometric Modules: Approaches that inject geometric priors via 3D foundation models or append mask decoders increase inference complexity and fail to foster intrinsic model capabilities.

Furthermore, while latent reasoning (encoding cues in continuous hidden space) shows promise, existing methods are limited to static scenes or rely on annotated reference images, hindering scalability to the dynamic, annotation-scarce video domain.

Methodology

The paper proposes 4DThinker, the first framework enabling VLMs to "think with 4D" by internally simulating scene evolution within the continuous hidden space using dynamic latent mental imagery. The framework consists of three core components:

1. Scalable, Annotation-Free Data Generation

To address the lack of training data, the authors introduce a pipeline that synthesizes 4D reasoning data directly from raw videos (specifically the SpatialVID dataset) without manual annotation.

Preprocessing: The pipeline uses MegaSaM for automatic camera trajectory estimation and SAM3 for object tracking to generate temporally consistent binary mask overlays. It identifies salient static landmarks (for camera motion analysis) and dynamic objects.
Motion-Centric QA Construction: The data is structured along two axes:
- Camera Motion: Generates Multiple-Choice Questions (MCQs) where static object displacement serves as visual evidence of camera movement.
- Object Motion: Formulates questions regarding direction, distance, speed, and spatial descriptions, grounded by bounding boxes derived from masks.
Imagery-Based CoT Synthesis: A high-level model generates Chain-of-Thought (CoT) reasoning traces that interleave textual analysis with dynamic mental imagery. These traces use placeholders for "imagery" frames, which are later replaced by latent tokens. The pipeline yields ~38K supervised samples and utilizes ~37K QA-only samples (from DSR-Train) for reinforcement learning.

2. Dynamic-Imagery Fine-Tuning (DIFT)

This stage grounds the model's intrinsic 4D latents in dynamic visual semantics through joint supervision.

Latent Representation: Visual encoder outputs for overlay images are compressed into $K$ latent visual tokens via partitioned mean pooling. These replace <imagery> placeholders in the CoT sequence.
Dual-Objective Loss: The model is trained with a combined loss:
- $L_{ce}$ : Standard causal language modeling loss on text tokens.
- $L_{sim}$ : A cosine-similarity loss enforcing alignment between the model's hidden state at a latent position and the ground-truth visual embedding.
Inference Mechanism: During inference, the model operates in a recurrent manner where the hidden state from the previous step serves as the input embedding for the next latent block, allowing the model to "imagine" the next frame of motion internally.

3. 4D Reinforcement Learning (4DRL)

To handle complex, compound motions (camera + object) that supervised data may not fully cover, the authors apply a modified Group Relative Policy Optimization (GRPO).

Outcome-Based Rewards: The reward function combines answer correctness ( $R_{acc}$ ) and adherence to the "think with 4D" format ( $R_{fmt}$ ).
Gradient Restriction: A critical modification restricts policy gradients to text tokens only, excluding latent positions. This prevents destabilizing noise caused by the mismatch between continuous latent propagation and discrete log-probabilities.

Key Contributions

4DThinker Framework: The first "think with 4D" framework that equips VLMs to mentally simulate 4D dynamics via latent visual tokens, enabling intrinsic reasoning about camera and object motion without external geometric modules.
Annotation-Free Pipeline: A scalable method to synthesize 4D reasoning data from raw videos, featuring CoT interleaved with dynamic mental imagery.
Two-Stage Training Recipe: A combination of DIFT (joint text-imagery supervision) for reasoning warm-up and 4DRL (text-only gradient optimization) for refining compound-motion reasoning.
Empirical Validation: Extensive experiments demonstrating consistent outperformance over strong baselines across multiple dynamic spatial reasoning benchmarks.

Results

The authors evaluated 4DThinker on DSR-Bench (fine-grained geometric measurement) and Dyn-Bench (holistic semantic understanding).

DSR-Bench: 4DThinker consistently improved performance across various base models (Qwen2.5-VL, Qwen3-VL, InternVL3.5). For instance, on Qwen3-VL-32B, the full pipeline achieved 62.0% accuracy, surpassing the previous state-of-the-art DSR Suite-Model (58.9%) and proprietary models like Gemini-2.5-Pro (31.7%). Notably, gains were most significant on absolute subtasks where base models performed near chance, suggesting the 4D latents provide necessary geometric grounding.
Dyn-Bench: The method achieved new SOTA results, with Qwen3-VL-32B reaching 75.4% average accuracy. The model demonstrated superior ability to disentangle ego-motion from object dynamics, a task where previous models often failed.
Ablation Studies: Removing the latent similarity loss ( $L_{sim}$ ) caused a significant performance drop (34.2% to 28.5%), confirming the necessity of visual alignment. Restricting RL gradients to text tokens was also shown to be crucial for stable optimization.

Significance and Claims

The paper claims that 4DThinker offers a new perspective on 4D reasoning by shifting from explicit, verbose textual descriptions or external geometric modules to intrinsic latent simulation. By treating dynamic scenes as "mental imagery" within the model's hidden space, the framework allows VLMs to reason about spatial-temporal evolution more naturally and efficiently. The authors assert that this approach validates that grounding Chain-of-Thought in continuous 4D imagery is more effective than purely textual or module-augmented approaches for dynamic spatial understanding.

Limitations Acknowledged:
The authors note that the current data pipeline relies on off-the-shelf geometric estimators (MegaSaM), meaning errors in these estimators may propagate into the training data, though 4DRL partially mitigates this. Additionally, the evaluation is currently limited to multiple-choice benchmarks, and extending the framework to open-ended generation tasks (e.g., embodied planning) remains a challenge.