Technical Summary: ViewSAM

Problem Statement

The paper addresses Cross-view Referring Multi-Object Tracking (CRMOT), a task that involves localizing and tracking multiple objects specified by natural language descriptions across synchronized camera views while maintaining globally consistent identities. While existing CRMOT methods achieve strong performance, they rely heavily on fully supervised learning paradigms requiring expensive, dense frame-level spatial annotations and cross-view identity correspondences. This reliance hinders the scalability of CRMOT to open-world scenes.

The authors propose exploring Weakly Supervised Cross-view Referring Multi-Object Tracking (WSCRMOT), where training data consists only of coarse-grained object category labels and raw multi-view videos, lacking spatial bounding boxes, cross-view IDs, or referring expressions. A preliminary empirical study reveals that directly applying foundation models like SAM2 and SAM3 to this task fails due to:

1. Lack of semantic understanding: Difficulty in interpreting complex referring expressions for long-term tracking.
2. Tracking bias: Models drift to distractors that partially match the text prompt, especially under occlusion.
3. Identity inconsistency: Failure to maintain global identity across views due to significant appearance and motion variations.

Methodology

The authors propose a two-stage framework that repurposes foundation models as pseudo-label generators and introduces a new architecture, ViewSAM, to learn view-aware cross-modal semantics.

Stage 1: Cross-view Pseudo Label Generation

Since explicit spatial and identity labels are unavailable, the framework generates high-quality pseudo labels using SAM3 and a Multimodal Large Language Model (MLLM).

• Affinity-guided Cross-view Re-prompting: SAM3 is first prompted with object category labels to generate single-view tracklets. An affinity-guided strategy then associates these tracklets across views. It computes tracklet-level embeddings and selects candidates with the highest affinity to an anchor. To refine these associations and correct noise caused by occlusion or distractors, a bi-directional re-prompting mechanism is employed. This uses an aggregated identity prototype to re-initialize SAM3 tracking in both forward and backward temporal directions, producing refined cross-view single-object trajectories.
• LLM-driven Referring-level Grouping: The refined trajectories are processed by an MLLM (Qwen3-VL-8B) to generate attribute-centric descriptions (capturing color, clothing, etc.). Trajectories with similar semantic descriptions are grouped under the same referring expression, forming the final CRMOT pseudo labels.

Stage 2: ViewSAM Architecture

ViewSAM is a CRMOT model built upon the SAM2 backbone, enhanced with approximately 10% additional trainable parameters to explicitly model view-aware cross-modal semantics.

• View-conditioned Cross-modal Alignment (VC-CMA): This module bridges the gap between view-variant visual observations and view-invariant textual expressions. It introduces a learnable dynamic View Token that interacts with visual and textual features via View-conditioned Cross Attention (VCCA). This allows the model to comprehend referring expressions by jointly reasoning over visual and textual modalities while accounting for view-specific priors.
• Candidate Generator: Extending SAM2 from single-object to multi-object tracking, this module generates box prompts to guide mask decoding. It is distilled from APTM and trained on ID-agnostic single-view pseudo labels.
• Bias-aware Recalibration (BAR): To mitigate the tracking bias where memory-guided decoding drifts to distractors, BAR compares memory-guided predictions with memory-free features. A learnable token assesses prediction reliability, and the model adaptively fuses the two predictions to refocus on objects consistent with the referring expression.
• Consistency-guided Cross-view Tracking Head (CGCT): This module projects tracklet representations into a shared view-invariant space. It utilizes the dynamic View Token to modulate features and employs consistency-guided objectives (intra-view and inter-view consistency losses) to enforce identity alignment across cameras without explicit cross-view identity supervision.

Key Contributions

1. First Weakly Supervised Framework: The paper proposes the first framework for WSCRMOT that significantly reduces reliance on dense annotations, establishing a baseline for future research in this setting.
2. Affinity-guided Re-prompting Strategy: A novel method for generating reliable cross-view pseudo labels by modeling cross-view affinity and iteratively refining trajectories.
3. ViewSAM Model: A SAM2-based model that explicitly models view-aware cross-modal semantics. It achieves robust referring understanding and consistent cross-view tracking with only ~10% additional parameters.
4. State-of-the-Art Performance: Extensive experiments demonstrate that ViewSAM achieves SOTA performance under weak supervision, substantially narrowing the performance gap with fully supervised methods.

Experimental Results

The framework was evaluated on the CRTrack benchmark under both in-domain and cross-domain settings, using metrics CVR-IDF1 (identity consistency) and CVR-MA (matching accuracy).

• Weakly Supervised Performance: ViewSAM outperformed other zero-shot and weakly supervised approaches (including SAM2 and SAM3 adapted with clustering) by significant margins (e.g., +3.36 in CVR-IDF1 and +3.55 in CVR-MA in in-domain evaluation).
• Comparison with Fully Supervised Methods: ViewSAM remained competitive with fully supervised methods like CRTracker. Notably, a fully supervised variant of ViewSAM further pushed performance boundaries, confirming the generalizability of the view-aware design.
• Ablation Studies: Removing key components (VC-CMA, Candidate Generator, BAR, or CGCT) resulted in significant performance drops, validating the necessity of view-conditioned alignment, bias recalibration, and consistency-guided tracking.
• Pseudo Label Quality: The affinity-guided association and bi-directional re-prompting strategies were shown to drastically improve pseudo label quality, leading to higher tracking accuracy.

Significance and Claims

The paper claims to establish a solid baseline for Weakly Supervised CRMOT, demonstrating that high-quality tracking can be achieved without dense spatial or identity annotations. By leveraging foundation models as pseudo-label generators and introducing a specialized architecture that treats view variations as learnable conditions rather than noise, the work enables scalable learning from abundant weakly labeled data. The authors emphasize that their approach naturally supports applications in multi-camera perception systems (e.g., intelligent surveillance, embodied AI) where collecting fine-grained annotations is prohibitively expensive. The work is presented as a preprint (arXiv:2605.02638v1), indicating it is a research contribution intended for the community to build upon.