In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems

1. Problem Statement

Multiple Object Tracking (MOT) is a fundamental computer vision task involving the localization of objects in every frame and the assignment of persistent identities to form continuous trajectories. While Single Object Tracking (SOT) follows one target, MOT must simultaneously handle multiple instances, often of the same class.

The paper identifies that despite significant progress, MOT remains challenging in real-world scenarios due to several interlinked failure modes:

• Occlusion: Objects being partially or fully hidden, leading to track fragmentation and identity switches.
• Detection Noise: False positives, missed detections, and score instability caused by cluttered backgrounds or camera motion.
• Identity Switching: Erroneous reassignment of IDs due to similar appearances or abrupt motion.
• Scale and Lighting Variations: Objects appearing at different sizes or under varying illumination conditions (e.g., night, low-light).
• Deployment Constraints: The need for real-time performance on edge devices, which conflicts with the high computational cost of modern deep learning models.
• Evaluation Gaps: Traditional metrics often fail to capture specific failure modes (e.g., motion accuracy in autonomous driving) or are saturated on standard benchmarks.

2. Methodology and Taxonomy

The authors organize the review not merely by algorithmic type, but by the challenges they target and the modeling paradigms they adopt. The paper synthesizes over 100 recent papers into the following key categories:

A. Detection and Target Association (Tracking-by-Detection)

• Concept: A per-frame detector provides candidates, and an association module links them across frames using appearance embeddings and motion priors (e.g., Kalman filters).
• Key Trends: Shift from hand-crafted features to learned embeddings (e.g., FairMOT, JDE). Recent work focuses on preserving low-score detections to handle occlusion (e.g., ByteTrack) and using hybrid training to bridge the gap between detection and association.
• Limitations: Highly dependent on detector quality; struggles with long-term occlusion and global reasoning.

B. Transformer-Based Approaches

• Concept: Uses attention mechanisms to model global spatial and temporal interactions. Queries represent objects, and cross-attention resolves correspondence.
• Key Trends: End-to-end architectures (e.g., TrackFormer, MOTR) that unify detection and tracking. Innovations include memory-augmented decoders for long-term persistence and "coopetition" label assignments to handle newborn objects.
• Limitations: High computational cost and data hunger; training instability when objectives compete.

C. Motion Models

• Concept: Relies on object dynamics and scene geometry (e.g., Kalman filters, LSTM, or learned predictors) to predict future states, especially when appearance is ambiguous.
• Key Trends: Integration of motion priors with appearance features; use of diffusion models or State Space Models (SSMs) for non-linear motion prediction.

D. Graph Models

• Concept: Represents detections/tracklets as nodes in a graph, casting association as edge prediction or global optimization.
• Key Trends: Message passing networks to learn global affinities; dynamic graphs that evolve with new observations; effective for multi-camera and multi-modal fusion.

E. Foundation Models

• Concept: Leverages large-scale pre-trained models (e.g., DINOv2, Grounding DINO, SAM) for robust feature extraction and open-vocabulary detection.
• Key Trends: Using language grounding for zero-shot detection and mask propagation (SAM2) for segmentation-aware tracking.

F. Emerging Paradigms

• Siamese Networks: Efficient matching via shared-weight encoders.
• Tracklet Association: Linking short-term fragments into long-term trajectories.
• Generative Tracking: Using diffusion models to denoise proposals and handle uncertainty in crowded scenes.
• Mamba-Based Tracking: Utilizing State Space Models (SSMs) for linear-complexity, long-horizon temporal reasoning.

3. Key Contributions

The paper makes five primary contributions to the field:

1. Comprehensive Overview: A concise synthesis of MOT's role in modern vision systems, covering both historical progression and state-of-the-art deep learning pipelines.
2. Challenge-Driven Synthesis: Instead of a simple list of algorithms, methods are categorized by the specific problems they solve (e.g., occlusion, motion ambiguity), providing a clear "challenge-to-solution" mapping.
3. Benchmark and Metric Analysis:
   • Datasets: Documents the shift from saturated pedestrian benchmarks (MOT17/20) to domain-specific, challenge-driven datasets (DanceTrack for motion, SportsMOT for high-speed, nuScenes/Waymo for 3D).
   • Metrics: Reviews classic metrics (MOTA, IDF1) and introduces newer, more nuanced metrics like HOTA (balancing detection and association), AMOTA (robust to confidence thresholds), and Motion-Centric metrics (VAE, VNE) for safety-critical applications.
4. Deployment Context: Highlights practical constraints (latency, hardware, safety) and analyzes how different architectures perform under these real-world conditions.
5. Future Directions: Outlines actionable research paths, including foundation model integration, open-vocabulary tracking, unified evaluation, and domain-adaptive methods.

4. Results and Findings

• Paradigm Shift: The field is moving from modular "detect-then-associate" pipelines toward end-to-end, global reasoning models (Transformers, Graphs) and generative approaches.
• Benchmark Saturation: Standard pedestrian benchmarks (MOT17/20) are showing metric saturation, where improvements are marginal. This has driven the community toward DanceTrack (appearance-agnostic, motion-heavy) and SportsMOT (high-speed, dense interaction) to better stress-test algorithms.
• Metric Evolution: MOTA is criticized for over-penalizing detection misses and under-penalizing ID switches. HOTA is emerging as the preferred metric for its balanced decomposition into Detection, Association, and Localization accuracy.
• Domain Specificity: No single "best" tracker exists. Performance varies drastically by domain:
  • Autonomous Driving: Requires 3D tracking, sensor fusion, and motion accuracy.
  • Sports: Requires handling rapid, non-linear motion and uniform appearances.
  • Biomedical: Requires lineage preservation and division event detection (Cell-HOTA).
• Foundation Models: While powerful for generalization and open-vocabulary tasks, they currently face challenges regarding inference latency and domain adaptation for real-time deployment.

5. Significance

This review is significant because it bridges the gap between theoretical algorithmic advances and practical engineering deployment.

• For Researchers: It provides a structured roadmap for future work, emphasizing the need for uncertainty quantification, robustness to domain shifts, and safety-centric evaluation.
• For Practitioners: It offers a pragmatic guide for selecting architectures based on specific constraints (e.g., using lightweight Siamese networks for edge devices vs. heavy Transformers for offline analysis).
• For the Community: It advocates for a shift away from "leaderboard chasing" on saturated datasets toward unified evaluation standards that reflect real-world complexities like long-term identity persistence, adverse weather, and multi-modal sensing.

In conclusion, the paper argues that the next phase of MOT research must move beyond isolated benchmark improvements to focus on system-level robustness, domain adaptability, and safety-critical reliability, leveraging foundation models and advanced architectural paradigms to solve persistent real-world challenges.