💻 computer science

STORM: Segment, Track, and Object Re-Localization from a Single Image

STORM is a unified framework that enables robust, reference-conditioned 6D object tracking from a single image by combining Hierarchical Spatial Fusion Attention for flexible conditioning with a learned verifier for automatic drift detection and re-localization, achieving superior accuracy and recovery from occlusions without requiring CAD models or manual intervention.

Original authors: Yu Deng, Teng Cao, Hikaru Shindo, Quentin Delfosse, Jiahong Xue, Kristian Kersting

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Deng, Teng Cao, Hikaru Shindo, Quentin Delfosse, Jiahong Xue, Kristian Kersting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to pick up a specific coffee mug from a cluttered kitchen counter. The robot has a single, perfect photo of that mug (the "reference"). Its job is to find that mug in a live video feed, even if the camera is shaking, the mug is half-hidden by a toaster, or the lighting changes drastically.

Most current robots are like students who memorize a single answer key. If the mug looks slightly different than the photo, or if it gets covered by a napkin, the robot gets confused, loses track, and keeps guessing blindly until it crashes or picks up the wrong thing. It doesn't know when it has made a mistake.

STORM (Segment, Track, and Object Re-Localization from a Single iMage) is a new system designed to fix this. Think of it as giving the robot a "smart assistant" that doesn't just look at the mug, but also constantly checks its own work.

Here is how STORM works, broken down into three simple parts:

1. The "Super-Spotter" (SOM)

The Problem: The robot's reference photo is clean and clear, but the live video is messy, dark, or full of other objects. Standard methods try to match the two images pixel-by-pixel, which fails when the view changes.

The STORM Solution: STORM uses a module called SOM (Segmenting Object Module). Imagine a detective who doesn't just look at a photo, but understands the story of the object.

Hierarchical Fusion: Instead of just comparing the mug photo to the video frame once, SOM looks at the video through multiple "lenses" (scales) and layers. It constantly asks, "Does this part of the video look like the mug in my photo?"
The Language Helper: If the robot is confused (e.g., there are two identical mugs), you can give it a text clue, like "the red mug." SOM uses this text to focus its attention, acting like a detective who says, "Ignore the blue mug; I'm looking for the red one."
The Result: It can draw a perfect outline around the mug, even if half of it is hidden behind a toaster.

2. The "Reality Check" (TOM)

The Problem: Even the best trackers eventually get lost. If the camera spins fast or the mug is fully covered, the robot might keep "tracking" a spot in the air where the mug used to be. It doesn't know it's wrong.

The STORM Solution: This is the paper's biggest innovation. STORM includes a module called TOM (Tracking Object Module). Think of TOM as a safety inspector or a "lie detector" for the robot's tracking.

The Memory Bank: TOM keeps a small "memory bank" of what the mug looked like when the robot was tracking it successfully just a second ago.
The Compatibility Test: For every new video frame, TOM asks: "Does what I see right now match the memory bank?"
The Energy Score: It gives a score. If the score is low, it means "Everything looks good." If the score gets high (like a rising alarm), it means "This doesn't look like the mug anymore; we are drifting!"
The Fix: Unlike other systems that just keep drifting, TOM detects this "drift" and immediately says, "Stop! We lost the target." It then triggers the "Super-Spotter" (SOM) to find the mug again from scratch.

3. The "3D Blueprint" (SAM3D)

To make sure the robot knows exactly where the mug is in 3D space (not just a flat 2D picture), STORM uses a tool called SAM3D.

Imagine taking the single reference photo and using it to build a rough, 3D clay model of the mug.
This 3D model acts as a rigid skeleton. Even if the robot can't see the whole mug, it can use this skeleton to guess the correct angle and position, ensuring the robot grabs the handle, not the bottom.

Why This Matters (According to the Paper)

The authors tested STORM on standard benchmarks (like the LM-O and YCB-Video datasets), which are like "final exams" for robot vision.

No Manual Help: STORM works without humans drawing boxes or masks on the video. It learns from just one reference image.
Better than the Rest: It performed better than previous top methods, especially in messy scenes where objects are hidden or moving fast.
Self-Correction: The most important result is that STORM can recover from mistakes. When the robot loses the object, STORM notices the failure and fixes it automatically, whereas other systems just keep failing silently.

In summary: STORM is a system that combines a smart visual search engine, a 3D builder, and a self-checking safety inspector. It allows a robot to track an object from a single photo, realize when it has lost track, and immediately find the object again, all without needing a human to step in and help.

Technical Summary: STORM (Segment, Track, and Object Re-Localization from a Single Image)

1. Problem Statement

Accurate 6D pose estimation and tracking are fundamental for physical AI systems, yet real-world deployment remains brittle. Current pipelines often rely on curated CAD models, manual masking, or per-object adaptation, failing under severe occlusion, motion blur, and rapid viewpoint changes. A critical limitation of existing methods is the lack of a principled mechanism to recognize tracking failure; trackers often operate "blindly," assuming the target remains within a local basin of attraction. When drift occurs due to distributional shifts between the canonical reference (e.g., a single clean image) and the observational query (e.g., a cluttered, occluded video frame), recovery is difficult without explicit failure detection.

2. Methodology

STORM (Segment, Track, and Object Re-Localization from a Single iMage) is a unified framework designed for annotation-free, reference-conditioned 6D pose tracking. It bridges semantic understanding and geometric consistency through two coupled, learnable modules: the Segmenting Object Module (SOM) and the Tracking Object Module (TOM). The framework leverages frozen foundation models (DINOv3, CLIP, SAM3D, FoundationPose) for representation and geometry, while training specific components for fusion and verification.

2.1. Segmenting Object Module (SOM)

The SOM performs coarse-to-fine feature alignment to recover object masks and 3D geometry from one or more reference images. Its core innovation is the Hierarchical Spatial Fusion Attention (HSFA) architecture:

Semantic Prior Injection: To resolve instance ambiguities (e.g., distinguishing a target mug from a distractor), STORM injects semantic constraints derived from Vision-Language Models (VLMs). A generic prompt and the reference image are processed by a VLM to generate a concise object descriptor, which is encoded by CLIP. This semantic embedding modulates normalized visual tokens via zero-initialized AdaLN/FiLM-style conditioning layers, allowing the model to learn residual feature-statistic corrections.
Hierarchical Spatial Fusion Attention (HSFA): Unlike simple concatenation or standard cross-attention, HSFA aggregates reference-view evidence and query-reference interactions across multiple scales. It constructs a coherent canonical representation from single or multi-view references, then aligns query features to this representation. This process is iterative, progressively refining the query representation with reference-conditioned evidence.
Implicit Alignment: HSFA induces dense token-to-token correspondence via attention mechanisms. The alignment matrix is interpreted as an alignment weight, through which a reference objectness prior is propagated to generate the query mask.
Geometric Anchoring: The aligned 2D features are lifted to SE(3) using SAM3D, which generates a canonical 3D mesh from the reference image(s). This mesh serves as a rigid geometric anchor, enabling the downstream pose estimator (FoundationPose) to map predicted masks onto metric 3D coordinates.

2.2. Tracking Object Module (TOM)

The TOM functions as a learned compatibility verifier to monitor tracking fidelity and trigger re-initialization when drift persists.

Compatibility Verification: TOM is trained as a binary classifier to distinguish between compatible observation-memory pairs (successful tracks) and incompatible pairs (identity confusion or drift). It takes a current observation crop and a memory pool of successful past crops as input.
Energy-Like Decision Rule: Instead of a hard binary decision, TOM outputs a continuous compatibility logit. This is converted into an energy-like score ( $E = -g_\theta$ ), where high energy indicates low compatibility (drift).
Temporal Smoothing and Thresholding: To avoid false positives from transient occlusions or motion blur, the energy score is smoothed using an Exponential Moving Average (EMA). Tracking loss is declared only if the smoothed energy exceeds a calibrated threshold for $L$ consecutive frames (e.g., $L=3$ ).
Automatic Re-localization: Upon detecting persistent drift, the system triggers the SOM to re-segment the object and re-initialize the pose, effectively closing the loop.

3. Key Contributions

Unified Framework: STORM proposes a single framework for annotation-free, reference-conditioned 6D pose tracking that addresses the distributional shift between reference and query data without requiring CAD models or instance-specific fine-tuning at inference time.
Hierarchical Spatial Fusion Attention (HSFA): A task-driven architecture for variable-view reference-conditioned segmentation. HSFA combines reference-view aggregation, query-reference cross-attention, and optional VLM-derived semantic conditioning to improve robustness under clutter and occlusion.
Tracking-Loss Verification: The formulation of tracking verification as a BCE-trained compatibility problem. Implemented via TOM, this mechanism uses an energy-like score derived from the compatibility logit to detect persistent drift and trigger automatic re-initialization.
Tracking Failure Benchmark: The establishment of a benchmark that evaluates trackers not only on pose accuracy but also on their ability to distinguish valid tracks from failure modes, demonstrating that TOM provides more reliable tracking-loss detection than fixed metric-learning baselines.

4. Experimental Results

Experiments were conducted on the LM-O and YCB-Video datasets under an annotation-free inference setting (no test-time masks, boxes, or object-specific updates).

Pose Estimation Accuracy: On LM-O, STORM achieved an ADD-AUC of 74.0% and ADD-S-AUC of 89.0%, outperforming the strong baseline (FoundationPose + CNOS) and approaching the performance of FoundationPose using ground-truth masks (78.0% and 93.0% respectively). On YCB-Video, STORM matched the ground-truth setup and exceeded the CNOS baseline by 5%.
Segmentation Quality: The SOM module achieved a mean Average Precision (mAP) of 67.7% across five BOP datasets, establishing a new state-of-the-art on four of them (LM-O, TUD-L, HB, YCB-Video) and outperforming supervised baselines on T-LESS without fine-tuning.
Tracking Robustness: In stress tests involving rapid motion and occlusion, STORM significantly outperformed a baseline without failure detection (ADD-AUC 74.64% vs. 52.76%). The TOM module successfully detected drift and triggered re-tracking, whereas the baseline failed to recover.
Efficiency: The system operates in real-time. The TOM verifier adds approximately 9.1ms per frame, keeping the steady-state tracking loop within the 25–33 FPS regime. SOM is invoked only for initialization or re-localization, running at ~22 FPS for single-reference settings.

5. Significance and Claims

The paper claims that STORM demonstrates that reference-conditioned representation learning, combined with explicit tracking-loss verification, outperforms traditional template matching in annotation-free settings. By integrating semantic understanding with geometric consistency, the framework reduces the labor-intensive requirements of deploying physical AI systems.

The authors emphasize that the ability to explicitly model the reliability of the tracking state is a key contribution. Unlike previous methods that drift silently, STORM provides a continuous self-checking signal that distinguishes transient uncertainty from persistent drift, enabling autonomous recovery. The work suggests that reconstructed geometry (via SAM3D) can be sufficient for practical pose registration if coordinate consistency is handled carefully, reducing the dependency on curated CAD assets.

The paper maintains a modest stance, acknowledging limitations such as ambiguity in symmetric or textureless objects (e.g., T-LESS) and the dependency on the fidelity of SAM3D reconstructions and the frozen FoundationPose backend. It notes that while the current zero-shot setting is adaptation-free, future work is needed for online threshold adaptation and handling category-disjoint novel objects.