💻 computer science

Efficient Tracking and Understanding Object Transformations

This paper introduces FluxGraph, a reactive object tracking method that significantly accelerates inference and improves performance over TubeletGraph by leveraging SAM2's multi-mask disagreement to detect transformations on-demand and eliminating the need to track all scene entities.

Original authors: Yihong Sun, Bharath Hariharan

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yihong Sun, Bharath Hariharan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a video of a busy kitchen. In the corner, a chef is slicing an apple. To a computer, this is a nightmare. The apple starts as one round object, then suddenly becomes a pile of slices, and the skin peels away. Most computer vision programs are like a student trying to memorize every single person in a crowded room, even if they only care about the one person cutting the fruit. They try to track everyone and everything in the video, frame by frame, which takes a massive amount of brainpower and time. This field, called "video object tracking," is all about teaching computers to follow moving things. But when those things change shape or break apart—like an apple being sliced or a butterfly leaving a cocoon—standard programs get confused and slow down. Understanding these changes is crucial for robots that need to cook, for wildlife cameras that watch animals grow, and for any smart system that needs to understand the real world.

Enter FluxGraph, a new method that acts like a clever detective instead of a hardworking scribe. The paper introduces this system as a way to track objects through these messy transformations much faster than previous methods, without losing accuracy. The authors observed that the leading method before this, called TubeletGraph, was doing too much "eager" work. It was like a security guard who checks every single door in a building every second, even if no one is moving. TubeletGraph tried to map out every object in the video all at once, which made it incredibly slow—taking about 4.4 seconds to process just one object for one frame.

FluxGraph changes the game by being "reactive." Instead of checking everything, it uses a trick built into a popular AI model called SAM2. SAM2 is like a guesser that sometimes isn't sure what it's looking at. When an object transforms (like the apple being cut), SAM2 gets confused and produces multiple different "guesses" (masks) for what the object might look like. FluxGraph watches for this confusion. If SAM2's guesses all agree, FluxGraph knows nothing interesting is happening and skips the heavy lifting. But if the guesses disagree, it knows a transformation is likely occurring and zooms in to investigate only that specific area.

The results are impressive. By only looking when the AI gets uncertain, FluxGraph is 3.3 to 10.7 times faster than TubeletGraph. On a specific test set called VOST, it cut the processing time down to about 1.33 seconds per object-frame, compared to TubeletGraph's 4.39 seconds. Even better, it didn't just get faster; it actually got slightly better at tracking. The authors found that by focusing only on the areas of uncertainty, they avoided tracking irrelevant objects, which reduced errors. They also replaced a complex, slow "semantic check" (which required a separate, heavy AI model to understand what objects were) with a simple "reverse tracking" trick. This new trick just asks: "If I follow this new piece of the object backward in time, does it lead back to the original object?" If yes, it's a real transformation; if no, it's just a random object that happened to be nearby.

In short, FluxGraph proves that you don't need to watch the whole movie to understand the plot twists. By paying attention only to the moments of confusion, it builds a clear map of how objects change—like a banana being peeled or a piece of foil being dispensed—while saving a huge amount of computing power. This makes it possible to run these complex tracking systems in real-time, opening the door for robots and smart cameras to understand the dynamic, changing world around them much more efficiently.

Technical Summary: FluxGraph – Efficient Tracking and Understanding Object Transformations

1. Problem Statement

Tracking objects through state transformations (e.g., slicing, peeling, phase changes) is critical for applications in robotic manipulation, wildlife monitoring, and video understanding. While the recent TubeletGraph framework successfully addresses the "Track Any State" task by simultaneously tracking objects through transformations and constructing a state graph of changes, it suffers from prohibitive computational costs.

The authors identify two primary bottlenecks in TubeletGraph that prevent real-time deployment:

Dense Entity Segmentation: It computes entity segmentation for every frame to account for any potential new object, regardless of whether a transformation is actually occurring.
Exhaustive Tracking: It tracks every entity in the scene, causing computational cost to scale with total scene complexity rather than the number of relevant transformations.

TubeletGraph employs an "eager" strategy, partitioning the entire video spatiotemporally upfront to ensure no transformation is missed, which results in significant overhead on frames and regions where no changes occur.

2. Methodology: FluxGraph

The authors propose FluxGraph, a reactive variant of TubeletGraph that eliminates the need for exhaustive partitioning by leveraging the internal uncertainty of the SAM2 (Segment Anything 2) model.

Core Insight

The central observation is that SAM2's internal multi-mask outputs implicitly encode where related objects (including transformation products) are likely to emerge. When SAM2 is uncertain about an object's location, its alternative mask hypotheses often diverge from the primary prediction. FluxGraph uses this multi-mask disagreement as a lightweight trigger for transformation detection.

Key Components

FluxGraph replaces TubeletGraph's two bottlenecks with two reactive mechanisms:

A. Reactive Candidate Discovery (via Multi-Mask Disagreement)
Instead of running entity segmentation on every frame, FluxGraph propagates the prompt object forward using SAM2 to generate a primary track $P$ and two alternative mask hypotheses $\{m_1, m_2\}$ .

Trigger Logic: The system skips downstream analysis if:
1. Low Disagreement: The alternative masks are nearly identical to the primary mask (indicating high confidence).
2. Small Residual Area: The differences between alternative masks and the primary mask fall entirely within regions already tracked by existing candidates.
Activation: Entity segmentation (via CropFormer) is invoked only when both conditions fail, indicating uncertainty in an untracked region. This limits segmentation queries to frames where a transformation is likely occurring.

B. Candidate Filtering via Reverse Tracking
To filter out false positives (e.g., unrelated objects that happen to appear in the disagreement region), FluxGraph replaces the semantic consistency check (which required an additional FC-CLIP model) with a purely geometric reverse tracking mechanism.

Process: For each candidate track generated from a disagreement region, the system tracks it backward in time from the current frame to the start of the video using SAM2.
Validation: The system computes the mean Intersection over Union (IoU) between the reverse track and the original prompt object track.
Decision: Candidates are retained only if the reverse track aligns with the original prompt (high IoU), confirming they are genuine products of the transformation. This removes the dependency on external semantic models.

3. Key Contributions

Bottleneck Identification: The paper characterizes the computational inefficiencies of TubeletGraph, specifically the cost of dense entity segmentation and exhaustive tracking.
FluxGraph Architecture: The proposal of a reactive framework that uses SAM2's internal multi-mask disagreement as a transformation trigger and reverse tracking for candidate filtering.
Performance Gains: Demonstrating that FluxGraph achieves significant speedups (3.3×–10.7×) across four benchmarks while matching or slightly improving tracking and state graph quality compared to TubeletGraph.

4. Experimental Results

The authors evaluated FluxGraph on four benchmarks: VOST, VSCOS, M3-VOS, and DAVIS17.

Efficiency:
- On VOST, FluxGraph reduces inference time from ~4.4s to ~1.33s per object-frame (a 3.3× speedup), increasing Masks Per Second (MPS) from 0.23 to 0.75.
- On DAVIS17 (a dataset with no transformations), the speedup is even more dramatic (10.7×), as the reactive trigger rarely fires, reducing overhead to nearly that of the base SAM2 tracker.
- Entity segmentation costs were reduced by an average of 9.8× across datasets.
Tracking Quality:
- FluxGraph matches or slightly exceeds TubeletGraph's performance. On VOST, it achieves a Jaccard score of 52.8 (vs. 50.9 for TubeletGraph) while running 3.3× faster.
- On VSCOS and M3-VOS, performance is within 0.6 J of TubeletGraph with no statistically significant gap.
State Graph Construction:
- FluxGraph preserves spatiotemporal recall (HST) and temporal recall (TR) comparable to TubeletGraph.
- It notably improves the semantic accuracy of resulting object descriptions (SO = 82.9 vs. 72.3 for TubeletGraph), attributed to tighter candidate proposals providing sharper visual prompts for the Vision-Language Model (VLM).
Ablation Studies:
- Removing reverse tracking causes a significant drop in tracking quality (e.g., J drops from 52.8 to 37.8 on VOST), confirming its necessity for filtering false positives.
- Removing reactive discovery (forcing dense segmentation) drastically increases inference time with negligible gains in tracking quality, validating the efficiency of the heuristic.

5. Significance and Limitations

Significance:
The paper claims that FluxGraph successfully decouples the cost of tracking from the complexity of the scene, scaling instead with the number of actual transformations. By utilizing the "uncertainty" inherent in foundation models like SAM2 as a signal for computation, FluxGraph enables real-time deployment of complex state-tracking tasks that were previously restricted to offline processing.

Limitations:
The authors acknowledge several constraints:

Fast Transformations: Transformations that occur too rapidly to generate significant multi-mask disagreement may evade the reactive trigger, leading to missed detections that TubeletGraph's exhaustive approach would catch.
Model Dependency: The method relies on SAM2's specific architecture (alternative heads producing meaningful hypotheses). Future trackers that do not expose internal uncertainty would require a different trigger mechanism.
Inherited Limitations: FluxGraph retains TubeletGraph's limitation of only detecting first-order transformations.
Ethical Considerations: The authors note that deployment in surveillance or human-monitoring contexts requires careful consideration of bystander rights, particularly given the ego-centric nature of the benchmark datasets.