ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking

Technical Summary: ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking

1. Problem Statement

Traditional Referring Multi-Object Tracking (RMOT) focuses on tracking targets based on explicit linguistic attributes (e.g., "the red car on the right"). However, these methods struggle with implicit constraints—instructions that require logical inference, social context, or causal reasoning to identify targets (e.g., "track the players of the team with better teamwork"). In real-world scenarios, human language is often nuanced, involving relationships, behaviors, and intentions that cannot be resolved through simple visual attribute matching. Existing models fail to generalize to these complex, high-level semantic instructions.

2. Key Contributions

The paper introduces three major contributions to bridge the gap between visual perception and logical reasoning in tracking:

• A New Task (ReaMOT): Defines Reasoning-based Multi-Object Tracking, a task requiring models to identify and track targets that satisfy implicit constraints via multi-step logical reasoning.
• The ReaMOT Challenge Benchmark: A large-scale, rigorous benchmark consisting of:
  • A dataset of 1,156 instructions categorized into High-Level Reasoning (social norms, causal logic, intentions) and Low-Level Perception (physical attributes, motion).
  • 423,359 image-language pairs across 869 diverse scenes and 21 object categories.
  • A tailored metric suite (RHOTA, RIDF1, RMOTA, RRcll, RPrcn) designed to evaluate both reasoning accuracy and tracking robustness.
• ReaTrack Framework: A novel, training-free framework that integrates the reasoning power of Large Vision-Language Models (LVLMs) with the temporal precision of advanced segmentation models.

3. Methodology (ReaTrack Framework)

ReaTrack operates through a synergistic three-module pipeline:

1. Reasoning-Aware Detection (RAD): Instead of standard detectors, it employs a Thinking-variant LVLM (e.g., Qwen3-VL-8B-Thinking). The model performs a "chain-of-thought" process to interpret complex instructions (e.g., analyzing a scoreboard to identify a winning team) and outputs bounding box coordinates for the identified targets.
2. Mask-Based Temporal Propagation (MBTP): To solve the temporal inconsistency inherent in frame-by-frame LVLM detection, the framework uses SAM2 (Segment Anything Model 2). It takes the bounding boxes from the previous frame as visual prompts to propagate segmentation masks and motion priors to the current frame, ensuring smooth tracking through occlusions and deformations.
3. Reasoning-Motion Association (RMA): This module reconciles the semantic detections (from the LVLM) with the motion priors (from SAM2) using Hungarian Matching based on IoU. It manages the trajectory lifecycle through four states:
   • Rectification: Correcting spatial drift in matched trajectories using LVLM detections.
   • Initiation: Starting new tracks for unmatched LVLM detections.
   • Extrapolation: Maintaining tracks during short-term occlusions using SAM2 predictions.
   • Termination: Removing tracks that remain unmatched beyond a set age.

4. Experimental Results

The framework was evaluated in a zero-shot setting on the ReaMOT Challenge:

• Superiority in Reasoning: ReaTrack achieved an RHOTA of 42.28% on the High-Level Reasoning subset, representing an approximately sixfold improvement over the strongest existing RMOT baseline (TempRMOT).
• Robustness: It consistently outperformed existing methods in both High-Level Reasoning and Low-Level Perception, demonstrating that the integration of "thinking" models and temporal propagators is essential for complex tasks.
• Ablation Insights: Experiments confirmed that the "Thinking" capability of the LVLM is critical for implicit instructions, while the MBTP module is vital for maintaining trajectory continuity (improving Recall significantly).

5. Significance

This work shifts the paradigm of multi-object tracking from purely visual recognition to semantic understanding. By providing a benchmark that demands logical inference, the authors pave the way for more intelligent autonomous systems (e.g., autonomous driving, intelligent surveillance, and robotics) that can interact with humans using natural, complex, and intention-based language.