STORM: End-to-End Referring Multi-Object Tracking in Videos

1. Problem Definition

Referring Multi-Object Tracking (RMOT) is the task of identifying, localizing, and tracking all objects in a video that semantically match a given natural language query (referring expression). Unlike traditional Multi-Object Tracking (MOT) which tracks all detectable objects, or Referring Single-Object Tracking (RSOT) which tracks only one target, RMOT requires the model to:

1. Ground multiple objects simultaneously based on complex linguistic descriptions (e.g., "the girl in the red dress and the dog running behind her").
2. Track these specific objects across time, maintaining their identities despite occlusion, scale changes, and motion.

Challenges:

• Data Scarcity: High-quality RMOT datasets are rare, noisy, and often lack diverse, unambiguous referring expressions.
• Modular Limitations: Existing approaches typically decouple grounding (detection) and tracking into separate modules (e.g., a detector + a tracker + a text encoder). This prevents the model from learning unified spatial-temporal representations and reasoning about causal or relational dependencies.
• Complexity: Current Multi-modal Large Language Models (MLLMs) often operate on static images or require external trackers when extended to video, failing to perform end-to-end reasoning.

2. Methodology: STORM

The authors propose STORM (Spatial-Temporal Object Referential Model), the first end-to-end MLLM designed specifically for RMOT.

A. Architecture

STORM adopts a LLaVA-style MLLM architecture:

• Visual Encoder: Uses a ViT-based encoder (specifically NaViT) to extract frame-based visual tokens. NaViT is chosen for its ability to handle arbitrary resolutions and aspect ratios without aggressive resizing, which is crucial for grounding tasks.
• Projector: A two-layer MLP maps visual tokens into the text embedding space.
• LLM Backbone: A LLaMA-based LLM receives the concatenated visual tokens and text query tokens. It autoregressively generates the output.
• Output Format: The model outputs structured plain text containing bounding box coordinates for each frame and object (e.g., Object 1: Frame 1: [x1, y1, x2, y2], Frame 2: ...). If an object is occluded, it outputs an empty box.

B. Task-Composition Learning (TCL) Strategy

To address the scarcity of RMOT data, the authors introduce Task-Composition Learning (TCL). Instead of training directly on limited RMOT data, the model learns the complex task by decomposing it into simpler, data-rich sub-tasks in a two-stage pipeline:

1. Stage 1: Pre-training on Sub-tasks

   • Image Grounding: Trained on large-scale referring image datasets (e.g., RefCOCO) to learn cross-modal alignment between text and spatial regions.
   • Single-Object Tracking (SOT): Trained on large-scale SOT datasets to learn temporal consistency and motion representations.
   • Referring SOT (RSOT): Trained on RSOT datasets to integrate visual-linguistic alignment with temporal tracking for a single target.
   • Goal: Equip the model with robust grounding and tracking capabilities before introducing multi-object complexity.

2. Stage 2: Fine-tuning with STORM-Bench

   • The model is fine-tuned on the proposed STORM-Bench dataset.
   • Chain-of-Thought (CoT) Training: The model is guided to generate intermediate reasoning traces (e.g., "I identify Object 1 at... Now I track...") before outputting the final coordinates. This encourages structured spatial-temporal reasoning.

3. Dataset: STORM-Bench

Recognizing that existing datasets (like LaMOT or Refer-KITTI) suffer from ambiguity and limited diversity, the authors constructed STORM-Bench.

• Scale: Contains 15,093 training videos and 714 evaluation videos, featuring 0.2M diverse referring expressions and 73.7K tracked objects.
• Annotation Pipeline (Bottom-Up):
  • Unlike top-down approaches that generate captions first and then boxes, STORM-Bench uses a bottom-up approach:
    1. Object-Level Captioning: Uses an MLLM with visual prompting (red bounding boxes) to generate detailed captions for individual objects.
    2. Verification: A second MLLM verifies if the caption uniquely matches the object using varied inputs (original, blurred background, cropped).
    3. Expression Synthesis: A reasoning LLM combines verified object captions to create complex multi-object expressions (e.g., conjunctions or shared attributes).
    4. Validation: Final verification ensures semantic consistency between the video, boxes, and the generated multi-object expression.
• Diversity: Covers 80 object classes (people, animals, furniture, electronics, etc.) and includes complex spatial-temporal relations, unlike datasets focused solely on pedestrians/vehicles.

4. Key Contributions

1. STORM Model: The first end-to-end MLLM for RMOT that unifies grounding and tracking without external detectors or trackers.
2. Task-Composition Learning (TCL): A novel training strategy that leverages abundant sub-task data (grounding, SOT, RSOT) to learn RMOT capabilities, significantly reducing the need for large-scale RMOT annotations.
3. STORM-Bench: A high-quality, large-scale dataset with accurate trajectories and diverse, unambiguous referring expressions, addressing the data bottleneck in the field.
4. State-of-the-Art Performance: Demonstrated superior performance across image grounding, single-object tracking, and RMOT benchmarks.

5. Experimental Results

The authors evaluated STORM on three levels of benchmarks:

• Image Grounding (RefCOCO/RefCOCO+/RefCOCOg): STORM achieved State-of-the-Art (SOTA) results, outperforming models like Shikra and M-GPT2. Notably, it achieved this without additional grounding data beyond RefCOCO, suggesting benefits from the video training.
• Referring Single-Object Tracking (Elysium): STORM outperformed the baseline Elysium model, particularly when tested with longer, more complex prompts (improving AUC by ~2.1%).
• Referring Multi-Object Tracking (STORM-Bench):
  • STORM significantly outperformed all baselines (Grounding DINO, Qwen2.5-VL, VisionLLMv2, LaMOT).
  • Metrics: Achieved 66.7 HOTA and 78.3 IDF1, surpassing the previous best (LaMOT) by +21.2 HOTA and +31.2 IDF1.
  • Ablation: The TCL strategy was proven critical; removing it and training only on RMOT data resulted in a massive performance drop (HOTA dropped from 66.7 to 34.8).

6. Significance and Impact

• Paradigm Shift: Moves RMOT from a modular, pipeline-based approach to a unified, end-to-end generative framework, allowing for coherent reasoning over appearance, motion, and language.
• Data Efficiency: The TCL strategy demonstrates that complex video tasks can be learned effectively by transferring knowledge from simpler, data-rich sub-tasks, mitigating the "data hunger" of MLLMs.
• Real-World Applicability: The ability to handle complex, relational queries (e.g., "the person to the left of the car") and maintain identity through occlusion makes this technology highly relevant for embodied AI, video understanding, and human-robot interaction.
• Resource Availability: The release of STORM-Bench and the code provides a necessary foundation for future research in referring multi-object tracking.