Technical Summary: Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced video understanding, they remain highly susceptible to hallucinations in dynamic scenes. The authors argue this failure stems from a lack of spatio-temporal monitoring: the ability to persistently track object identities, states, and relations over time.

Current evaluation benchmarks suffer from two critical limitations that obscure this deficit:

1. Local Cue Reliance: Many questions can be resolved by inspecting a single frame or short segment, allowing models to bypass global temporal reasoning.
2. Final-Answer Obscurity: Evaluation based solely on a final answer masks the reasoning process. A model may produce a correct answer through statistical priors or coincidental correctness without genuinely resolving the underlying spatio-temporal dependencies.

Consequently, existing benchmarks provide a limited view of a model's true understanding capabilities, failing to distinguish between genuine temporal reasoning and "shortcut learning."

2. Methodology

The paper proposes a two-pronged approach: a new benchmark for rigorous evaluation (STEMO-Bench) and a novel object-centric framework for video understanding (STEMO-Track).

2.1 STEMO-Bench: Spatio-Temporal Monitoring Benchmark

STEMO-Bench is designed to explicitly evaluate an MLLM's ability to track object-centric dynamics. It introduces a structure-first data construction pipeline and a fine-grained evaluation protocol.

• Data Construction:
  • Expert-Driven Fact Annotation: Instead of generating QA pairs directly from raw video, the authors first construct a human-verified structural representation. A pool of trained annotators identifies entities, assigns canonical identifiers, and records observable states and relations as structured tuples (e.g., (t, object, state)).
  • Identity Binding: Annotators manually trace entities across temporal anchors to ensure consistent identity tracking, even through occlusion or state changes.
  • Automatic QA Generation: An LLM generates complex target questions based on these verified facts, ensuring they require multi-step reasoning across time.
  • Decomposition: Each target question is decomposed into a sequence of atomic sub-questions corresponding to key steps in spatio-temporal monitoring (e.g., identifying an object, retrieving its state at a specific time, verifying a transition).
• Evaluation Metrics:
  • Target Accuracy ($A_{target}$): Standard accuracy on the final question.
  • Sub-question Accuracy ($A_{sub}$): Accuracy across all prerequisite sub-questions.
  • Consistency ($A_{cons}$): The proportion of correctly answered sub-questions given that the target answer was correct. A high $A_{target}$ with low $A_{cons}$ indicates reliance on spurious correlations rather than robust monitoring.

2.2 STEMO-Track: Object-Centric Framework

To address the failure modes exposed by STEMO, the authors propose STEMO-Track, a pipeline that explicitly models entities and their temporal evolution rather than treating video as an unstructured sequence of frames.

• Pipeline Architecture:
  1. Temporal Chunking: The video is partitioned into disjoint temporal chunks.
  2. Chunk-wise State Extraction: A VLM-based state extractor ($E_\phi$) processes each chunk independently to identify visible objects and estimate their states, producing local observations.
  3. Temporal Aggregation: A temporal aggregation module ($A_\psi$) consolidates local observations into global, consistent object trajectories. This involves a four-step identity-linking algorithm:
     • Similarity Scoring: Matching objects across chunks based on visual attributes and spatial proximity.
     • Constraint Filtering: Enforcing temporal proximity and confidence thresholds.
     • Conflict Resolution: Using bipartite matching to ensure one-to-one identity mapping.
     • Trajectory Generation: Merging linked observations into coherent trajectories ($S_o$).
  4. Query-based Trajectory Retrieval: For a given query, a retriever ($R_\omega$) selects the relevant subset of trajectories ($S_Q$) needed to answer the question.
  5. Trajectory-Guided Answering: An answer generator ($G_\eta$) reasons over the retrieved trajectories to produce the final answer.

This design separates representation construction from query-specific reasoning, allowing the video representation to be constructed once and reused for multiple questions.

3. Key Contributions

1. STEMO-Bench: A benchmark featuring human-verified object-centric facts that evaluates intermediate reasoning via atomic sub-questions, distinguishing genuine temporal understanding from coincidental correctness.
2. STEMO-Track: A novel object-centric pipeline that systematically constructs and queries structured object trajectories, explicitly grounding answers in retrieved temporal data.
3. Systematic Evaluation: A comprehensive evaluation of modern MLLMs on target versus intermediate consistency, highlighting spatio-temporal deficiencies and demonstrating the efficacy of explicit monitoring.

4. Experimental Results

The authors evaluated STEMO-Track against proprietary end-to-end MLLMs (e.g., Gemini-3.1-Pro, GPT-5), open-weight models (e.g., Qwen3-VL, InternVL3), and structured pipelines (e.g., VideoMind, TraveLER) on STEMO-Bench.

• Performance: STEMO-Track (using Gemini-3-Flash) achieved 77.1% Target Accuracy, significantly outperforming the strongest proprietary baseline (Gemini-3.1-Pro at 61.4%) and open-weight models.
• Consistency: STEMO-Track demonstrated superior Consistency ($A_{cons}$) at 81.3%, compared to 69.2% for Gemini-3.1-Pro. This indicates that STEMO-Track's correct answers are more likely to be supported by faithful intermediate reasoning.
• Statistical Significance: Improvements were statistically significant ($p < 0.005$) via paired bootstrap tests.
• Ablation Studies:
  • Visual Budget: Reducing the visual budget for the answer generator had minimal impact on STEMO-Track (Gemini) but severely degraded open-weight models, suggesting the trajectory memory effectively encodes temporal cues.
  • Aggregation: Deterministic concatenation of trajectories outperformed LLM-based summarization, preventing the omission of critical low-salience details.
• Qualitative Analysis: Case studies revealed that baseline models often answered target questions correctly while failing prerequisite sub-questions (e.g., misidentifying an object's identity), confirming that target-only metrics overestimate true understanding.

5. Significance and Claims

The paper claims that the primary limitation of current video MLLMs is the lack of explicit spatio-temporal monitoring. By introducing STEMO-Bench, the authors provide a tool to rigorously diagnose this failure, moving beyond "coincidental correctness."

The significance of STEMO-Track lies in its object-centric formulation, which aligns the model's internal computation with the requirements of video understanding: maintaining persistent identities and tracking state evolution. The authors assert that explicit trajectory construction is a key factor in reducing hallucinations and improving reasoning consistency. They conclude that future work should focus on improving fine-grained visual attribute binding within these trajectories to further enhance robustness, rather than relying solely on end-to-end scaling.