💻 computer science

StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videos

The paper introduces StateTrace, an object-centric framework that equips VideoLLMs with a structured spatiotemporal memory to explicitly reason about hidden object states during invisible intervals in long videos, significantly improving performance on a newly proposed HSR-Bench and existing benchmarks.

Original authors: Yu Han, Wenhao Li, Yichao Cao, Hongyan Xu, Shuo Yang, Shan You, Xiu Su

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Han, Wenhao Li, Yichao Cao, Hongyan Xu, Shuo Yang, Shan You, Xiu Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, machines have become remarkably skilled at watching videos and describing what they see. They can identify a person running, a car turning, or a dog chasing a ball. However, these digital observers have a significant blind spot when it comes to the unseen. If an object disappears from the screen—hidden behind a wall, tucked inside a box, or covered by another object—current AI models often forget it exists entirely. They treat "invisible" as if it were "unknown," unable to reason that the object is still there, just out of sight, and that its state (like its location or color) likely hasn't changed. This limitation becomes a major hurdle when analyzing long videos, where objects may remain hidden for extended periods, yet a viewer might still need to know where they are or what happened to them.

To solve this problem, a team of researchers has developed a new approach called StateTrace. Instead of relying solely on what is visible in the current frame, this system gives the AI a way to remember and track objects even when they are hidden. The researchers built a framework that acts like a persistent memory for the video, recording not just what is seen, but also what happens when things disappear. It tracks the moment an object is covered, the reason it vanished, and the last known details about its position and appearance. By organizing this information into a structured timeline, the system can answer questions about the hidden state of an object long after it has left the screen.

The researchers tested this idea by creating a new set of challenges specifically designed to trap standard AI models. They compiled a benchmark called HSR-Bench, which contains over 1,400 video questions derived from nearly 1,400 unique videos. These questions focus on scenarios where objects are occluded, contained, or covered, requiring the AI to infer their status without direct visual proof. For instance, a question might ask about the color of a car that was blocked by a bus for several seconds, or which cup holds a set of nails after they were swapped between containers. When the researchers applied StateTrace to powerful existing video models, the results were striking. On this new benchmark, the models improved their accuracy dramatically, with some jumping from roughly 40 percent correct to over 64 percent. This suggests that giving an AI a structured way to reason about the invisible is a crucial step toward making it understand the world as humans do.

The core of the StateTrace system is a three-step process that happens before the AI even attempts to answer a question. First, the system analyzes the entire video offline, breaking it down into segments. It identifies objects, tracks their movements, and notes exactly when and how they disappear. If a car is covered by a bus, the system records that event, noting the bus as the cause and the car's last known position. It builds a detailed map of these events, creating a "spatiotemporal state memory" that links objects to their history of visibility and interaction. This memory is not just a list of frames; it is a connected graph that understands relationships, such as one object being inside another or one blocking the view of another.

When a user asks a question, the system does not simply scan the video for matching words. Instead, it consults this memory map to find the specific moments relevant to the query. If the question asks about a hidden object, the system retrieves the last time that object was visible and the events that led to its disappearance. It then summarizes this history into a concise narrative, explaining the object's journey and its current likely state. This summary is combined with the actual video clips and fed into the AI model. By providing this explicit context, the model is no longer guessing; it is reasoning based on a reconstructed timeline of events. The system essentially tells the AI, "The object you are looking for is not visible right now, but here is exactly where it was last seen and what happened to it."

The researchers found that this method works best when the video is long and the object is hidden for a significant time. In tests involving videos ranging from a few seconds to an hour long, the StateTrace framework consistently outperformed other methods that simply tried to compress the video or retrieve relevant clips. The improvement was most dramatic in cases of heavy occlusion, where objects were almost completely blocked from view. For example, in one test case, a standard model failed to identify the color of a car hidden behind a bus, incorrectly guessing based on the final visible frame. With StateTrace, the model correctly recalled the car's color from before it was blocked, maintaining the continuity of the scene despite the visual gap.

To ensure their findings were robust, the team conducted a series of experiments to see which parts of their system were most important. They discovered that the ability to reason about why an object disappeared was critical. If the system only knew an object was gone but not whether it was covered, moved away, or hidden inside something, its performance dropped significantly. Similarly, the step that summarized the spatial relationships and movement history proved to be the most vital component. Without this summary, the AI struggled to make sense of the raw data, much like a person trying to solve a puzzle with pieces scattered on the floor rather than assembled into a picture. The system also showed that simply feeding the whole video to the AI was not enough; the targeted retrieval of specific, relevant moments was essential for accuracy.

The success of StateTrace highlights a fundamental shift in how video understanding can be approached. For years, the focus has been on making models see better or process more frames. This work suggests that the next leap forward comes from making models think better about what they cannot see. By treating the invisible state of an object as a solvable puzzle rather than a missing piece of data, the researchers have enabled AI to handle the complexities of real-world observation. The results indicate that with the right kind of memory and reasoning, machines can begin to bridge the gap between what is shown and what is known, a capability that is essential for any system intended to navigate the long, continuous flow of real life.

Technical Summary: StateTrace

Problem Definition

Current Vision-Language Models (VLMs) have demonstrated strong capabilities in general video understanding tasks such as captioning and temporal localization. However, they exhibit a fundamental weakness in hidden-state spatiotemporal reasoning, particularly in long videos. When a target object becomes invisible due to occlusion, containment, or covering, existing models often fail to infer its state, mistakenly equating "invisible" with "unknown." They rely heavily on explicit visual evidence at the query time rather than modeling the latent state of objects during prolonged invisible intervals. This limitation prevents them from answering questions about an object's location or attributes after it has disappeared from view, even if the context implies its state should persist.

Methodology: StateTrace

To address this, the authors propose StateTrace, an object-centric framework designed to endow VideoLLMs with an explicit mechanism for hidden-state reasoning. The framework operates through three distinct stages:

1. Offline Spatiotemporal State Memory Construction

Before inference, the system performs an offline parsing pass to construct a reusable, structured memory ( $M$ ) organized as a chunk-level directed graph ( $G_V$ ) with an auxiliary entity access graph ( $G_E$ ). This memory consists of three components:

Semantic Context ( $M_{sem}$ ): Chunk-level attributes including entities, actions, scenes, and subtitles.
Spatial Location Memory ( $M_{loc}$ ): Frame-level absolute geometry (normalized centroid coordinates and area) and pairwise spatial relations (e.g., left, right, near, far) between objects. Relations are stored only when they change to reduce redundancy.
Spatial Event Memory ( $M_{evt}$ ): Records of visibility transitions (appearance/disappearance) and, crucially, disappearance-cause interpretation. For non-boundary disappearances, a VLM module analyzes short video contexts to infer causes such as "inside" (containment), "occluded," or "other."

The parsing process involves dividing the video into chunks, performing object grounding and mask propagation (using SAM2) to maintain cross-chunk continuity, and extracting spatial relations and event metadata.

2. Question-Driven Spatiotemporal Evidence Summarization

At inference time, given a question $q$ , the system retrieves relevant evidence from the state memory rather than processing the entire video raw.

Retrieval: The question is converted into a structured query to retrieve relevant video chunks via a two-stage process (recall based on entity/semantic matching, followed by reranking with temporal priors).
Summarization: The retrieved sub-memory is compressed into two complementary summaries:
- Semantic Aggregation Summary ( $g_q$ ): Condenses answer-relevant textual evidence and sub-questions.
- Spatial Summary ( $s_q$ ): Constructs structured spatiotemporal evidence by merging frame-level timelines, repairing visibility states, and extracting anchor events (e.g., occlusion, entry, reappearance). This summary explicitly tracks object state evolution across invisible intervals.

3. Summary-Augmented Answer Generation

The final stage combines the retrieved visual evidence (video chunks), subtitle context, and the generated semantic and spatial summaries ( $g_q, s_q$ ) to form a multimodal input for the VideoLLM. The model then generates the answer based on this enriched context, enabling explicit reasoning about why an object disappeared and how its state evolved while invisible.

Key Contributions

Problem Formulation: The paper systematically defines hidden-state spatiotemporal reasoning as the capability to infer a target's latent state during prolonged invisibility based on prior interactions and state changes, identifying this as a core missing capability in current VideoLLMs.
StateTrace Framework: A novel object-centric framework that shifts long-video reasoning from purely direct visual evidence to structured, state-centric reasoning over latent object dynamics. It introduces a reusable spatiotemporal state memory and a mechanism for explicit disappearance-cause modeling.
HSR-Bench: The introduction of HSR-Bench, a diagnostic benchmark containing 1,427 video-QA samples from 1,384 unique videos. It specifically targets object persistence and latent-state inference under occlusion, containment, and long-delay querying, scenarios often under-evaluated in existing benchmarks.

Experimental Results

The authors evaluated StateTrace across multiple public benchmarks (MLVU, VideoMME, LongVideoBench) and the new HSR-Bench using various VideoLLM backbones (InternVL2.5, VideoLLaMA3, Qwen2.5-VL, LLaVA-Video).

Public Benchmarks: StateTrace consistently improved performance across all backbones. Notably, on LongVideoBench, it showed significant gains, suggesting its effectiveness in maintaining state continuity over extended temporal spans. For example, VideoLLaMA3-7B improved from 59.8 to 64.5 on LongVideoBench.
HSR-Bench: The improvements were most substantial here, where hidden-state reasoning is the primary challenge. StateTrace improved base model performance by 15.14 to 28.31 points. For instance, VideoLLaMA3-7B achieved a score of 64.19 on HSR-Bench (up from 39.59), and Qwen2.5-VL-3B improved from 29.85 to 57.74.
Ablation Studies:
- Removing disappearance-cause reasoning degraded performance, indicating that knowing why an object disappeared is crucial for state recovery.
- Removing spatial summary generation caused the largest performance drop, highlighting its role as a critical interface for converting low-level records into explicit state evolution accounts.
- Removing key clip retrieval also reduced performance, confirming the value of focusing on relevant temporal contexts.
Occlusion Severity: Analysis on HSR-Bench showed that StateTrace provided the largest gains under heavy occlusion, where hidden-state reasoning is most critical.

Significance and Claims

The paper claims that StateTrace addresses a specific and critical gap in long-video understanding: the inability of current models to reason about objects that are temporarily invisible. By explicitly modeling the causes of disappearance and maintaining a structured state memory, StateTrace allows models to infer that an object's state persists even when it is not visible. The authors position this not just as an incremental improvement in accuracy, but as a shift in reasoning paradigm—from evidence-driven to state-centric. The introduction of HSR-Bench is presented as a necessary step to diagnose and drive progress in this specific capability, which existing benchmarks fail to adequately evaluate. The work demonstrates that explicit state modeling is essential for robust spatial reasoning in complex, long-horizon video scenarios.