CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking

Technical Summary: CD-RMOT-Bench: Benchmarking the Cross-Domain Referring Multi-Object Tracking

1. Problem Definition

The paper addresses Cross-Domain Referring Multi-Object Tracking (CD-RMOT), a novel and challenging problem that extends Referring Multi-Object Tracking (RMOT) to scenarios involving significant visual domain shifts. While standard RMOT aims to locate and track specific objects based on natural language expressions within a single visual distribution (in-domain), CD-RMOT evaluates whether a model trained on a labeled source domain can reliably follow language expressions in an unlabeled target domain with different visual conditions (e.g., weather changes, viewpoint shifts, or synthetic-to-real gaps).

The core challenge identified is that domain shifts do not merely degrade object detection; they critically destabilize the temporal association and target selection processes conditioned on language. Existing RMOT models often fail to maintain semantic fidelity when the underlying visual distribution changes, leading to the selection of trajectories that are visually plausible but inconsistent with the referring expression.

2. Methodology

A. CD-RMOT-Bench (The Benchmark)
To enable systematic study, the authors construct CD-RMOT-Bench, a unified benchmark integrating three complementary resources under a shared evaluation protocol:

• Refer-KITTI-V2: Serves as the real clear-domain anchor, providing natural-language expressions and trajectories from real driving scenes.
• Refer-vKITTI: The controlled digital-twin core. It is constructed by mapping Refer-KITTI-V2 annotations to aligned Virtual KITTI 2 scenes. This process involves:
  • Per-scene identity mapping: Establishing one-to-one correspondences between real and synthetic objects based on geometric consistency and temporal overlap.
  • Annotation transfer: Preserving the natural language expression while rewriting target IDs and video names to match the synthetic domain.
  • Residual Quality Control: Filtering ambiguous cases and ensuring target visibility.
  • Coverage: Supports 10 controlled domains (clear, rain, fog, morning, overcast, sunset, and three viewpoint shifts: 15°L/R, 30°L/R).
• Refer-BDD: A real adverse-domain anchor based on BDD100K, providing realistic domain variations (e.g., adverse weather) for synthetic-to-real and real-to-synthetic transfer evaluation.

The benchmark supports four evaluation modes: weather transfer, viewpoint transfer, synthetic-to-real transfer, and real-to-synthetic transfer.

B. QCA (Query-Centric Adaptation)
The authors propose QCA, a reference adaptation framework designed to stabilize the query space that bridges visual trajectories and referring expressions. QCA operates on the temporal object queries produced by a shared RMOT backbone and introduces three mechanisms to address specific failure modes:

1. Cross-Domain Temporal Mean Teacher (CDT-MT): Addresses target-query instability by enforcing consistency between a student model and an Exponential Moving Average (EMA) teacher model on reliable target-domain queries.
2. Adaptive Curricular Domain Adversarial (ACDA): Aligns task-relevant source and target query representations using domain adversarial learning. It focuses on queries with high "task-relevance" (product of objectness and referring logits) and utilizes the source/target split as natural domain labels.
3. Language-Guided Continuous Adaptation (LGCA): Mitigates language-conditioned semantic drift. It uses the referring expression as a semantic anchor (via a pooled text embedding) to refine query features, ensuring selected trajectories remain consistent with the language input over time.

The total loss function combines the supervised source-domain RMOT loss with consistency, adversarial, and language-guided adaptation losses.

3. Key Results

Extensive experiments on CD-RMOT-Bench yield the following findings:

• Severe Performance Degradation: Domain shifts cause significant drops in RMOT performance. For instance, in the controlled vKITTI clear → fog setting, the state-of-the-art TempRMOT model's HOTA drops from 28.69 (in-domain) to 16.00.
• Root Cause Analysis: The degradation is not solely due to detection errors (DetA drops) but is driven more critically by failures in temporal association and target selection (AssA drops). Models select trajectories that are visually plausible but semantically inconsistent with the language query.
• Limitations of Existing Methods: Adapted Referring Video Object Segmentation (RVOS) baselines (e.g., MUTR, ReferDINO) perform poorly under the RMOT trajectory protocol, indicating that mask-based segmentation does not directly solve the identity-aware tracking problem under domain shift.
• Effectiveness of QCA: The proposed QCA framework establishes a strong baseline, significantly recovering performance across all transfer settings (weather, viewpoint, synthetic-real). For example, in vKITTI clear → fog, QCA achieves a HOTA of 21.37 compared to 16.00 for TempRMOT. However, performance remains below the in-domain reference, highlighting the difficulty of the problem.

4. Significance and Contributions

The paper claims the following contributions:

• Problem Pioneering: It introduces CD-RMOT as a critical new problem, highlighting the need for robust visual-language tracking under domain shifts, a gap largely overlooked by existing in-domain RMOT studies.
• Benchmark Construction: It presents CD-RMOT-Bench, the first unified benchmark for CD-RMOT. Its unique value lies in combining controlled digital-twin variations (isolating weather/viewpoint effects) with realistic synthetic-real transfer, allowing for a principled disentanglement of domain shift effects from semantic content changes.
• Methodological Baseline: It provides QCA, a query-centric adaptation framework that demonstrates the necessity of stabilizing language-conditioned queries. The results validate that future research must address query stability, domain alignment, and language-guided correction jointly.

The authors conclude that while QCA offers a strong baseline, the substantial performance gaps remaining in CD-RMOT-Bench indicate that robust language-guided tracking across visual domains remains an open and challenging direction for the community.