💻 computer science

LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue

This paper introduces LMM-Track4D, a novel framework that enhances Large Multimodal Models' 4D dynamic reasoning capabilities through a new trajectory-grounded dialogue task and benchmark (Track4D-Bench), utilizing specialized components like Ray-Time Geometry Encoding and an Object-Slot Kinematic decoder to achieve robust 3D state estimation under occlusion and viewpoint variations.

Original authors: Chaoyue Li, Yongxue Xu, Jie Feng, Jiayu Ding

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaoyue Li, Yongxue Xu, Jie Feng, Jiayu Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a basketball game on TV, but instead of just seeing the action, you are having a conversation with a super-smart AI about it.

The Problem: The "Forgetful" AI
Current AI models (Large Multimodal Models) are great at looking at a single photo or a short video clip and saying, "That's a player dribbling." But if you ask them a follow-up question a few seconds later, like, "Where did that player go after he passed the ball?" or "Can you draw the exact path the ball took while it was hidden behind another player?", they often get lost.

Think of current AIs like a tourist who takes a photo of a street, then closes their eyes. If you ask them, "What happened next?" they have to guess based on the single photo. They don't have a continuous "mental movie" running in their head. They struggle to keep track of objects moving through time, especially when those objects get blocked from view (occlusion) or the camera angle changes.

The New Task: "Trajectory-Grounded Dialogue"
The authors of this paper created a new game to test this specific weakness. They call it Trajectory-Grounded Multi-Turn Spatiotemporal Dialogue.

Here is how the game works:

The Scene: You show the AI a video clip (like a basketball play).
The Chat: You ask a question: "Who scored the three-pointer at 0.8 seconds?"
The Twist: The AI must not only answer with words but also provide a structured 3D map of where that player and the ball were at every single moment, even if they were hidden behind someone else.
The Follow-up: You ask another question based on the first answer: "Now, show me the path the ball took from 0.5s to 1.2s."

The AI has to act like a referee who never loses track of the ball, even when it's behind a player, and can draw its exact path in 3D space.

The Solution: LMM-Track4D
To solve this, the team built a new AI system called LMM-Track4D. They gave it three special "superpowers" to keep its mental movie running smoothly:

The "Geometry Compass" (RTGE):
- Analogy: Imagine a GPS that doesn't just know "where" you are on a map, but also knows the exact angle of the sun and the time of day.
- How it works: This part of the AI injects "Ray-Time Geometry" into the video. It teaches the AI to understand not just the pixels, but the 3D shape of the room and the exact time each frame happened. This helps the AI understand depth and perspective, not just flat images.
The "Memory Token" (TRK):
- Analogy: Think of this as a sticky note the AI writes on and keeps in its pocket.
- How it works: When the AI sees a player, it writes a note about that player's identity and movement. When the conversation moves to the next turn, instead of starting from scratch, it pulls out that sticky note. This allows the AI to remember, "Oh, that's Player #6, and he was moving right," even if the camera angle changes or the player is briefly hidden. It keeps the story continuous.
The "Stable Sketcher" (OSK-RA Decoder):
- Analogy: Imagine trying to draw a smooth line on a shaky boat. Most AIs draw a wobbly, jagged line. This decoder is like a gimbal stabilizer that keeps the line smooth.
- How it works: This part of the system takes the AI's observations and turns them into a clean, smooth 3D path. It uses a "residual anchor" technique, which means it starts with a rough guess of where the object is and then makes tiny, precise adjustments to fix any errors, ensuring the path doesn't jump around wildly.

The Test: Track4D-Bench
To prove their system works, they built a new test called Track4D-Bench.

It contains 526 video clips (mostly from sports and traffic scenes) with over 7,500 labeled objects.
It's like a final exam where the AI has to answer questions and draw 3D paths simultaneously.
The test checks if the AI can handle tricky situations like objects disappearing behind cars (occlusion) or answering questions about the same object over a long period.

The Results
When they ran the test:

Old AIs: Even the smartest existing models (like GPT-4o or specialized video AIs) could answer the text questions okay, but they failed miserably at drawing the 3D paths. They often lost track of the object or gave up when it was hidden.
LMM-Track4D: This new system was the clear winner. It maintained a consistent "mental movie" of the scene. It could tell you exactly where the ball was at 0.8 seconds and draw its smooth path even when it was blocked by a player.

The Big Takeaway
The paper concludes that to make AI truly understand the 4D world (3D space + time), we can't just make the AI "bigger" or feed it more videos. We need to build specific tools (like the Memory Token and Geometry Compass) that force the AI to explicitly track the state of objects as they move. It's the difference between a camera that takes snapshots and a director who keeps a continuous script of the movie.

Technical Summary: LMM-Track4D

Problem Statement
Recent Large Multimodal Models (LMMs) have demonstrated significant capabilities in image and video understanding. However, they struggle with sustained 4D dynamic reasoning across time, viewpoints, and multi-turn interactions. Current models are optimized to interpret already observed content but lack mechanisms to preserve explicit dynamic state. Consequently, while they may recognize isolated observations, they fail to maintain a physically coherent account of evolving scene states as new evidence arrives, particularly when dealing with occlusions, identity continuity, and long-horizon motion trends.

Methodology: LMM-Track4D
To address this gap, the authors propose LMM-Track4D, a framework designed to maintain and update dynamic state throughout interaction. The system is built upon a Qwen3.5-9B backbone with Low-Rank Adaptation (LoRA) and integrates three core trainable modules:

RTGE (Ray–Time Geometry Encoding): Inserted between the visual encoder and the multimodal backbone, RTGE injects explicit geometric and temporal information into visual tokens. It computes 6-D Plücker descriptors (combining ray direction and moment) from camera intrinsics/extrinsics and applies Fourier embeddings for time. These are added to patch features, allowing the model to ground visual tokens in 3D space and time.
TRK (Streaming State Token): A persistent state-memory token that acts as a "target slot." Instead of re-inferring target identity and motion from raw history at every turn, the TRK token accumulates evidence across dialogue turns. Its hidden state is propagated from turn $k$ to $k+1$ , ensuring continuity of identity and motion dynamics.
OSK-RA (Object-Slot Kinematic, Residual-Anchor) Decoder: This module handles the final 3D trajectory decoding. It separates target selection (semantic query) from geometric refinement (kinematic query). The decoder scores evidence rays attached to visual tokens, fuses them, and computes a coarse 3D anchor via weighted line intersection. A residual head then refines this anchor to produce the final 3D coordinates. This anchor-plus-residual design mitigates first-step bias accumulation.

Benchmark: Track4D-Bench
The authors introduce Track4D-Bench, a benchmark constructed through a structured pipeline linking verified 3D trajectories, visual evidence, and dialogue.

Scale: 526 clip-level dialogue samples (428 short, 98 long) spanning 23.5k frames and 7.5k object annotations, sourced from APIDIS, KITTI, and WildTrack.
Task: Trajectory-grounded multi-turn spatiotemporal dialogue. Models must answer qualitative spatiotemporal questions and, when required, output structured 3D trajectories over short clips or local segments of longer interactions.
Evaluation: Uses a unified dialogue-turn protocol. Language quality is measured via CIDEr, BLEU-4, and METEOR. Geometric accuracy is measured via $SAcc@0.5$ (point accuracy), $Traj-Acc@1.0$ (trajectory accuracy), and $TAcc@0.5$ (temporal point accuracy), all in world coordinates.

Key Contributions

Task Formulation: Identified the limitation of current LMMs in sustained 4D reasoning and formulated "trajectory-grounded multi-turn spatiotemporal dialogue" to study it.
Benchmark: Introduced Track4D-Bench, providing a trainable, measurable, and diagnostically interpretable testbed for 4D dynamic reasoning with both short-clip and long-clip regimes.
Model Architecture: Proposed LMM-Track4D, which integrates RTGE, persistent TRK state propagation, and OSK-RA decoding to preserve and update dynamic information across views, time, and dialogue turns.
Empirical Evidence: Demonstrated that explicit dynamic state modeling is a useful design principle for eliciting 4D reasoning, showing consistent improvements over strong baselines in dense non-keyframe inference, occlusion recovery, and long-range identity preservation.

Results
Experiments on Track4D-Bench reveal a significant performance gap between generic LMMs and LMM-Track4D:

Baseline Performance: Strong proprietary (e.g., GPT-4o) and open-source models (e.g., Qwen3-VL, LLaVA-Video) achieve non-trivial language scores but fail to provide comparable structured 3D outputs. Even Same-dataset SFT controls (fine-tuned baselines) remain well below LMM-Track4D.
LMM-Track4D Performance: Achieved the best results across all metrics, including $0.758$ $SAcc@0.5$, $0.626$ $Traj-Acc@1.0$, and $0.619$ $TAcc@0.5$.
Ablation Studies:
- Removing OSK-RA caused the largest drop in step-wise fidelity ($TAcc@0.5$ dropped from 0.619 to 0.425).
- Removing RTGE resulted in the second-largest degradation in geometric metrics.
- Removing TRK (or using the backbone only) eliminated stable structured geometry outputs entirely, confirming the necessity of persistent state propagation.
History Dependence: Analysis showed that "No-history" settings perform worst, especially in late turns, while "Self-history" (using the model's own TRK state) consistently outperforms it, approaching the "Gold history" (oracle) upper bound.

Significance and Claims
The paper claims that eliciting 4D dynamic reasoning in LMMs is not merely a matter of scaling video perception or generic video QA. Instead, it requires task formulations, supervision, and architectures that explicitly model evolving scene dynamics over time. The results suggest that explicit dynamic state modeling serves as an effective inductive bias for this capability.

The authors remain modest regarding the current limitations: the setting depends on structured supervision, the scale is limited compared to general video datasets, and the temporal horizon and openness of interaction are constrained. However, the work establishes that maintaining physically coherent state across multi-turn interactions is a distinct and solvable challenge with the right architectural inductive biases.

Technical Summary: LMM-Track4D

More like this