AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Technical Summary: AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Problem Statement
Audio-visual speaker tracking aims to localize and track active speakers by leveraging synchronized auditory and visual cues. While essential for applications like intelligent video editing, surveillance, and human-computer interaction, existing benchmarks fail to rigorously evaluate models in complex, real-world scenarios. Current datasets are predominantly limited to static cameras, single-instance scenes, or short, trimmed clips (5–10 seconds) with coarse annotations. These oversimplified settings bias evaluation toward static audio-visual co-occurrence rather than testing robust spatiotemporal modeling and cross-modal reasoning under dynamic conditions such as visual occlusion, camera motion, and relative position changes. Consequently, existing Audio-Visual Instance Segmentation (AVIS) methods struggle to generalize to the fragmented, asynchronous, and human-centric dynamics found in real-world environments.

Methodology
To address these limitations, the authors introduce AVTrack, a novel benchmark designed specifically for human-centric AVIS in complex scenes, and AVTracker, a modular baseline framework.

• The AVTrack Dataset:

  • Scope: AVTrack consists of 871 video clips containing 3,120 densely annotated instance tracklets. It is released exclusively as a test set to serve as a stable, long-term reference for measuring progress, decoupling evaluation from dataset-specific training.
  • Complexity: The dataset is curated to enforce eight specific challenging conditions: Visual Occlusion, Relative Position Change, Background Switch, Camera Motion Change, Multiple Instances, Multi-turn Sounding, Audio-Visual Inconsistency, and Instance Scale Dynamics.
  • Diversity: Unlike previous datasets limited to laboratory settings or single sources (e.g., TV series), AVTrack spans diverse genres including films, vlogs, interviews, and stage performances.
  • Annotation: The dataset features pixel-level mask annotations with cross-frame identity consistency, constructed via a four-stage pipeline involving automatic pre-annotation (Grounded-SAM), human-in-the-loop verification, and rigorous quality control.

• The AVTracker Baseline:

  • Architecture: AVTracker employs a "divide-and-conquer" strategy structured as a three-stage modular framework:
    1. Speaker Chunks Aggregation: Utilizing Whisper for transcription and ECAPA-TDNN for speaker embeddings, the system aggregates fragmented speech segments into coherent "speaker chunks" based on speaker similarity.
    2. Local Window Process: For each chunk, a Local Reasoner (Qwen3-VL) conditions on speech transcripts and visual observations to associate utterances with visible persons. SAM3-Video generates candidate masks, which are aligned with the reasoner's bounding box predictions via Intersection-over-Union (IoU) to form local tracklets.
    3. Global Window Process: A Global Reasoner aggregates local tracklets across the entire video to resolve cross-chunk identity associations, producing coherent instance-level trajectories.
  • Design Philosophy: The framework is designed for extensibility, allowing the integration of advanced components (e.g., speech separation via MossFormer2) without architectural overhaul.

Key Contributions

1. AVTrack Benchmark: The introduction of a challenging, human-centric AVIS dataset that covers eight complex conditions significantly more demanding than existing benchmarks (e.g., AVISeg, AVA-ActiveSpeaker).
2. Performance Gap Identification: Comprehensive evaluations reveal substantial performance degradation for state-of-the-art Video Instance Segmentation (VIS) and AVIS methods on AVTrack, highlighting the inadequacy of current models in complex real-world settings.
3. AVTracker Baseline: The proposal of a simple yet effective, modular baseline that significantly outperforms existing methods by leveraging structured audio-visual reasoning and long-term identity association.

Results
Experiments on the AVTrack benchmark demonstrate a significant gap between current methods and the proposed baseline:

• VIS Methods: Purely visual approaches (e.g., VITA, LBVQ) yield extremely low HOTA scores (below 12.0), indicating visual cues alone are insufficient for human-centric tracking in these scenarios.
• AVIS Methods: Existing AVIS models (e.g., AVISM, ACVIS) incorporating audio information improve HOTA scores to approximately 20–21, but still fall short of robust performance.
• AVTracker Performance: AVTracker achieves a HOTA of 29.08, outperforming the strongest AVIS baseline by over 8 points. It also achieves superior scores in Association Accuracy (AssA: 28.47) and MOTA (16.20).
• Ablation Studies: The results confirm that sufficient model capacity in both speech processing and visual reasoning (VLM) is critical. Furthermore, the use of dynamic windowing and speaker chunk aggregation significantly contributes to performance, while speech separation (MossFormer2) offers marginal gains only when separation quality is high.
• Qualitative Analysis: AVTracker demonstrates robustness in handling relative position changes and multi-turn sounding, whereas end-to-end models often fail under visual occlusion or when multiple speakers are present.

Significance and Claims
The paper posits that AVTrack serves as a rigorous stress test for the field of human-centric audio-visual understanding. The authors claim that the observed performance degradation of current models on AVTrack underscores the need for more robust spatiotemporal modeling and cross-modal reasoning capabilities. By providing a challenging benchmark and a modular baseline, AVTrack aims to foster research into next-generation models capable of fine-grained reasoning in dynamic, real-world environments. The authors emphasize that AVTrack is released strictly as a research benchmark to study model limitations and encourage responsible, privacy-preserving development, rather than to enable real-world surveillance. They suggest future directions include agentic reasoning with memory mechanisms and improved handling of audio-visual inconsistencies.