💻 computer science

Tracking and Understanding Object Transformations

This paper introduces the "Track Any State" task and the VOST-TAS benchmark to address the challenge of tracking objects through state transformations, proposing TubeletGraph, a zero-shot system that recovers lost tracks and generates semantic state graphs to describe evolving object dynamics.

Original authors: Yihong Sun, Xinyu Yang, Jennifer J. Sun, Bharath Hariharan

Published 2026-01-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yihong Sun, Xinyu Yang, Jennifer J. Sun, Bharath Hariharan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie where a whole apple gets sliced into pieces, or a caterpillar turns into a butterfly. If you were a standard movie camera operator, you might lose track of the "apple" the moment it gets cut, because suddenly you aren't looking at one red round thing anymore; you're looking at five white chunks. You might also lose the "caterpillar" because it disappears inside a shell, only for a butterfly to appear later.

This is the problem the paper "Tracking and Understanding Object Transformations" tries to solve. The authors, from Cornell University, argue that current computer vision systems are like bad movie directors: they can follow a character as long as they look the same, but if the character changes their costume, breaks apart, or transforms, the system loses them completely.

Here is a simple breakdown of their solution, TubeletGraph, and how it works.

The Problem: The "Missing Person" in the Crowd

Current tracking systems rely heavily on appearance. They say, "I see a red apple; I will follow that red apple." But when the apple is cut, the red skin is gone, and the white flesh is different. The system thinks, "The red apple is gone!" and stops tracking. It doesn't realize that the white pieces are the apple, just in a new state.

The authors noticed a specific pattern: these systems usually make false negatives. They think an object has vanished when it has actually just changed shape.

The Solution: TubeletGraph (The "Detective with a Net")

The authors built a system called TubeletGraph. Think of it as a detective who doesn't just follow one person, but casts a wide net to catch everyone in the scene, then uses logic to figure out who is who.

Here is how it works in three steps:

1. Casting the Net (The "Tubelets")

Instead of just following the specific object you asked about (like the apple), TubeletGraph splits the entire video into tiny, moving chunks called "tubelets."

Analogy: Imagine a video is a river. Instead of just following one specific leaf (the apple), the system puts a net in the water that catches every leaf, twig, and stone that appears.
It tracks the original apple, but it also starts tracking new things that appear later, like the apple slices or the butterfly that emerges.

2. The Detective's Logic (Proximity and Semantics)

Now the system has a "soup" of many different tracks. It needs to figure out which new tracks belong to the original object. It uses two rules:

The "Hug" Rule (Spatial Proximity): If the apple is cut, the pieces will be right next to where the apple was. If a new object appears far away (like a hand holding a knife), it's probably not part of the apple. The system checks: "Is this new piece close to the original apple?"
The "Identity" Rule (Semantic Consistency): The system asks, "Does this new thing make sense as a version of the old thing?"
- Good match: A slice of apple looks like apple flesh.
- Bad match: A hand holding the apple looks like a hand, not an apple.
- Analogy: If you are looking for your dog, and you see a golden retriever nearby, you say, "That's my dog!" But if you see a cat nearby, you say, "No, that's not my dog," even if the cat is right next to you.

3. The Storyteller (The State Graph)

Once the system recovers the missing pieces (the apple slices or the butterfly), it doesn't just keep tracking them; it asks a powerful AI (a Large Language Model) to explain what happened.

It looks at the "before" (whole apple) and the "after" (slices).
It asks the AI: "What happened here?"
The AI answers: "The apple was cut into slices."
The system then draws a map (a State Graph) showing the timeline: Apple -> Cut -> Slices.

What They Achieved

The paper introduces a new challenge called "Track Any State." This isn't just about following an object; it's about following an object through its changes and describing those changes.

To test this, they created a new dataset called VOST-TAS, which contains videos of things changing (like peeling a banana or slicing a tomato) with detailed labels.

The Results:

Better Tracking: Their system, TubeletGraph, is better at keeping track of objects that change shape than existing top-tier systems (like SAM2). It recovers the "missing" pieces that other systems drop.
Better Understanding: It doesn't just draw a box around the object; it generates a text description of the transformation (e.g., "The caterpillar emerged from the chrysalis").
Zero-Shot: The system didn't need to be retrained on every specific type of transformation (like "cutting" vs. "peeling"). It figured it out on the fly using general knowledge.

Summary

In short, the paper says: "Current computers lose track of things when they change. We built a system that casts a wide net to catch everything, uses logic to figure out which new things are actually the old things in disguise, and then asks an AI to write a story about how the transformation happened."

This allows computers to understand that a "butterfly" and a "chrysalis" are part of the same story, and that "apple slices" are still "apple," even if they don't look like the original fruit anymore.

Technical Summary: Tracking and Understanding Object Transformations

Problem Definition: Track Any State

The paper addresses a critical limitation in current computer vision systems: the inability to track objects through significant state transformations (e.g., an apple being sliced, a butterfly emerging from a cocoon). Existing object tracking methods, which rely heavily on appearance consistency, typically fail when an object's geometry or texture changes drastically, often resulting in "false negative" errors where the tracker loses the target. Conversely, existing state-change recognition methods often lack the spatiotemporal grounding to track the resulting objects after a transformation.

To bridge this gap, the authors introduce Track Any State, a new task defined as: given a video and an initial object prompt (a mask in the first frame), the system must:

Track the object consistently through its transformation, including all resulting parts (e.g., tracking both the apple core and the slices).
Detect and Describe the state changes, outputting a structured representation of the transformation (time, action verb, and resulting objects).

Methodology: TubeletGraph

The authors propose TubeletGraph, a zero-shot framework designed to recover missed objects and reason about transformations without task-specific fine-tuning. The pipeline consists of three main stages:

1. Spatiotemporal Partitioning (Tubelet Generation)

To overcome the search space problem of finding a "missing" object after a transformation, the system creates a dense spatiotemporal partition of the video:

Initial Partition: The first frame is segmented into entities using CropFormer, including the user-provided prompt mask.
Tracking: Each initial entity is tracked forward in time using SAM2.
Dynamic Initiation: As the video progresses, the system identifies "empty regions" (pixels not covered by existing tracks). When an entity is detected in these regions, a new track (a "tubelet") is initiated.
Result: This creates a "soup" of tubelets covering nearly all pixels in the video, ensuring that post-transformation objects (which appear as new connected components) are captured as candidate tracks.

2. Candidate Recovery via Reasoning

Not every new tubelet is a part of the transformed target object (e.g., a hand entering the frame). The system filters candidates using two priors:

Spatial Proximity: Based on the assumption that transformed objects remain near the original location, the system calculates the overlap between a candidate track and the multiple candidate masks predicted by SAM2 for the original prompt.
Semantic Consistency: Based on the assumption that identity semantics are preserved (e.g., a chrysalis becomes a butterfly, not a bird), the system uses masked CLIP features to compute semantic similarity between the prompt track and the candidate track.
Selection: Only tubelets satisfying both proximity and semantic thresholds are added to the final tracking set.

3. State Graph Construction

Once a new valid track is identified, the system treats its emergence as a marker for a state transformation:

Querying VLMs: The system queries a multi-modal Large Language Model (GPT-4.1) with the initial frame and the frame where the new track appears.
Generation: The LLM is prompted to describe the action verb (e.g., "cut," "unroll") and the identities of the resulting objects.
Graph Output: These descriptions are parsed to construct a State Graph, a structured representation linking pre-transformation tracks, the transformation event, and post-transformation tracks.

Key Contributions

Task Formulation: Introduction of Track Any State, a unified task combining object tracking with state-change detection and description.
Dataset: Release of VOST-TAS, a new benchmark dataset derived from the VOST validation set, containing 57 video instances, 108 annotated transformations, and 293 resulting object annotations with temporal boundaries and semantic descriptions.
System: Development of TubeletGraph, a zero-shot system that recovers missed objects via spatiotemporal partitioning and generates state graphs using vision-language reasoning.

Experimental Results

The authors evaluate TubeletGraph on multiple benchmarks, including VOST, VSCOS, M3-VOS, and DAVIS 2017.

Tracking Performance: TubeletGraph achieves state-of-the-art (SOTA) performance on VOST and VSCOS, outperforming base SAM2 and its fine-tuned variants. Notably, it improves the Jaccard index (J) by 2.5 points over base SAM2 and surpasses fine-tuned SAM2 in the last 25% of video frames ( $J_{tr}$ ), demonstrating superior robustness to transformations.
State Graph Quality: On the VOST-TAS benchmark, the system achieves a temporal localization precision of 43.1% and a semantic accuracy of 81.8% for action verbs. While the overall recall for fully correct transformations (temporal + semantic + spatial) is modest (6.5%), the authors note this reflects the significant difficulty of the unconstrained task.
Ablation Studies: The system is shown to be robust to hyperparameter variations. The ablation confirms that the spatiotemporal partition is crucial for candidate retrieval, while semantic and spatial constraints are necessary to filter false positives.

Significance and Claims

The paper positions TubeletGraph as a step toward more robust and informative vision systems for downstream applications. The authors claim:

Zero-Shot Capability: Unlike methods requiring extensive domain-specific fine-tuning, TubeletGraph operates without training on the target transformation types, offering better generalization.
Comprehensive Representation: By combining tracking with state graph generation, the system provides a complete representation of object evolution, which is useful for tasks like action-grounding (e.g., robots understanding pre- and post-conditions of sliced apples), video editing, and scene modeling.
Error Recovery: The system explicitly addresses the "false negative" bias in existing trackers by recovering missed objects and using them to reason about the transformation that caused the loss.

The authors acknowledge limitations, including high computational costs (approx. 7 seconds per frame) and potential error propagation from the base tracker (SAM2) or the VLM. They conclude that while real-time application is currently limited, the system is highly valuable for offline analysis, such as generating training annotations for robotics or analyzing biological development.