💻 computer science

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

The paper introduces OneTrackerV2, a unified multimodal visual tracking framework that employs a Meta Merger and Dual Mixture-of-Experts (DMoE) architecture to enable efficient end-to-end training across diverse modalities, achieving state-of-the-art performance on multiple benchmarks while maintaining robustness and inference efficiency.

Original authors: Lingyi Hong, Jinglun Li, Xinyu Zhou, Kaixun Jiang, Pinxue Guo, Zhaoyu Chen, Runze Li, Xingdong Sheng, Wenqiang Zhang

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lingyi Hong, Jinglun Li, Xinyu Zhou, Kaixun Jiang, Pinxue Guo, Zhaoyu Chen, Runze Li, Xingdong Sheng, Wenqiang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific dog in a video. Sometimes you only have a normal color camera (RGB). Other times, you might have a thermal camera (seeing heat), a depth camera (seeing distance), or even a text description saying "a dog."

The Problem with Old Trackers
Until now, if you wanted to track that dog using different types of cameras, you had to build a separate "brain" for each one.

Need to track with just color? Build a color brain.
Need to track with heat? Build a heat brain.
Need to track with depth? Build a depth brain.

This is like hiring a different specialist for every single tool you own. It's expensive, slow to train, and if you lose one tool (like your thermal camera breaks), that specific brain becomes useless. Even worse, if you tried to combine them into one big brain, the information would get messy, like trying to listen to a symphony where the drums and violins are playing the same notes at the same time—it creates noise and confusion.

The Solution: OneTrackerV2
The authors introduce OneTrackerV2, a single, unified "super-brain" that can handle any combination of cameras (or even just one) all at once. It learns everything in one go, rather than needing separate training sessions for each camera type.

Here is how it works, using simple analogies:

1. The "Universal Translator" (Meta Merger)

Imagine you have a team of people speaking different languages (Color, Heat, Depth). If you just throw them all in a room and tell them to talk, they might talk over each other.

OneTrackerV2 uses a module called Meta Merger. Think of this as a Universal Translator or a Conductor.

It takes the raw data from whatever cameras you have (even if one is missing!) and translates them all into a single, shared language.
This ensures that the "color" data and the "heat" data can work together smoothly without fighting each other.
The Magic: If you lose the thermal camera, the translator doesn't panic. It just focuses on the color data and keeps the conversation going. This makes the system very robust.

2. The "Specialized Experts" (Dual Mixture-of-Experts)

Once the data is translated, it goes into the main processing unit. The authors realized that tracking a moving object requires two very different skills:

Motion Tracking: Figuring out where the object is going (speed, direction, time).
Identity Tracking: Figuring out what the object looks like based on its specific sensor (is it hot? is it deep?).

If you mix these two skills in one big brain, they get confused. To fix this, OneTrackerV2 uses Dual Mixture-of-Experts (DMoE). Imagine a high-end restaurant kitchen with two specialized stations:

The Motion Chef (T-MoE): This station only cares about movement. It ignores whether the food is hot or cold; it just focuses on the speed and direction of the ingredients.
The Flavor Chef (M-MoE): This station only cares about the specific "flavor" of the input (the sensor type). It focuses on the unique details of the thermal or depth data.

Why separate them?
In the past, one big chef tried to do both, leading to "feature conflict" (confusion). By separating them, the "Motion Chef" gets really good at predicting movement, and the "Flavor Chef" gets really good at recognizing specific sensor details. They work side-by-side but don't get in each other's way.

3. The "Smart Manager" (Router Clustering)

How does the system know which chef to call? It uses a Router.

If the object is moving fast, the Manager sends the data to the Motion Chef.
If the data is from a thermal camera, the Manager sends it to the Flavor Chef who specializes in heat.
The paper shows that this Manager learns to be very specific: it doesn't just guess; it learns exactly which "expert" is best for which situation.

The Results

The paper claims that this new system is a game-changer because:

One Size Fits All: It works for 5 different types of tracking tasks (Color, Color+Depth, Color+Heat, etc.) using the exact same code and settings.
Better than Specialists: Surprisingly, this "generalist" brain is actually better at tracking with just color than the old "specialist" brains that were designed only for color.
Resilient: If a camera fails (missing modality), the system keeps working almost as well as before.
Efficient: Even when they shrink the model down to make it faster (compression), it still performs incredibly well, beating other fast models.

In Summary
OneTrackerV2 is like a Swiss Army Knife that is actually better at being a knife than a dedicated pocket knife, and better at being a screwdriver than a dedicated screwdriver. It uses a Universal Translator to clean up the input and Specialized Chefs to handle movement and identity separately, resulting in a tracker that is faster, smarter, and more reliable than anything that came before it.

Technical Summary: OneTrackerV2 – Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

1. Problem Statement

Visual object tracking (VOT) is typically categorized by input modality into RGB tracking and RGB+X tracking (incorporating Depth, Thermal, Event, or Language data). Existing approaches suffer from significant limitations in scalability, efficiency, and robustness:

Separated Trackers: Design and train task-specific architectures for each modality combination, leading to high computational costs and lack of generalization.
Adaptation-Based Trackers: Fine-tune pretrained RGB trackers for downstream multimodal tasks. This multi-stage training often results in suboptimal convergence and relies on task-dependent weights.
Preliminary Unified Trackers: Attempt to handle multiple modalities in a single model but often rely on naive token concatenation. This leads to feature entanglement, where heterogeneous information (spatio-temporal motion vs. modality-specific patterns) conflicts within a shared space, causing optimization interference.
Vulnerability to Missing Modalities: Most unified models fail or degrade significantly when specific input modalities (e.g., Depth or Thermal) are absent during inference, as they rely on fixed input configurations.

2. Methodology: OneTrackerV2

The authors propose OneTrackerV2, a unified framework designed to handle diverse multimodal tracking tasks through a single architecture, shared parameters, and one-step end-to-end training. The core architecture consists of three primary components:

2.1 Meta Merger

To address the challenge of integrating heterogeneous inputs without feature entanglement, the authors introduce the Meta Merger.

Function: It maps RGB and auxiliary modality features (e.g., Depth, Thermal) into a unified embedding space using a learnable meta embedding ( $F_{meta}$ ).
Mechanism: Instead of direct concatenation, the meta embedding acts as a global mediator. It interacts with enhanced RGB and auxiliary features ( $F'_{rgb}$ and $F'_{x}$ ) via lightweight convolutional transformations.
Benefit: This creates a compact, modality-agnostic semantic representation. Crucially, because fusion is centered on the meta embedding rather than fixed input concatenation, the module remains robust even when specific modalities are missing.

2.2 Dual Mixture-of-Experts (DMoE)

Following the Meta Merger, the unified features are processed by a Vision Transformer backbone equipped with a Dual Mixture-of-Experts (DMoE) module. Unlike standard MoE designs that focus on a single objective, DMoE explicitly decouples two distinct learning tasks:

T-MoE (Temporal-MoE): Dedicated to capturing complex spatio-temporal matching patterns and motion cues.
M-MoE (Multimodal-MoE): Dedicated to fusing multimodal knowledge and handling modality-specific features.
Shared Expert: A common expert ( $E_{shared}$ ) that processes general representations.

Decoupling Mechanisms:
To prevent T-MoE and M-MoE from collapsing into overlapping patterns, the authors employ two specific loss functions:

Expert Decoupling Loss ( $L_{dis}$ ): Penalizes the cosine similarity between the outputs of T-MoE ( $y_T$ ) and M-MoE ( $y_M$ ), forcing them into orthogonal functional subspaces.
Router Cluster Loss ( $L_{cluster}$ ): Enforces the M-MoE router to produce consistent expert selections for samples from the same modality while maintaining discriminative routing across different modalities. This ensures M-MoE learns modality-specific abstractions rather than generic features.

2.3 Optimization Strategies

Stochastic Modality Perturbation: During training, the model employs random modality replacement (swapping RGB and auxiliary inputs) and random modality masking. This exposes the Meta Merger to diverse configurations, enhancing robustness to missing data and preventing over-reliance on specific sensors.
Loss Function: The total objective combines standard tracking losses (classification, IoU, L1) with the proposed decoupling and clustering losses.

3. Key Contributions

Unified Framework: Introduction of OneTrackerV2, a single framework that supports diverse multimodal tracking tasks (RGB, RGB+D, RGB+T, RGB+E, RGB+N) via one-step training with shared parameters, eliminating the need for task-specific architectures or multi-stage fine-tuning.
Meta Merger Module: A novel module that aggregates heterogeneous features into a unified space, enabling efficient cross-modal interaction while maintaining robustness under missing-modality conditions.
Dual Mixture-of-Experts (DMoE): A structural solution to heterogeneous objective conflicts that decouples spatio-temporal relation modeling from multimodal feature embedding. This expands the representational space and improves performance with negligible computational overhead.
Comprehensive Evaluation: Demonstration of state-of-the-art (SOTA) performance across 5 tracking tasks and 12 benchmarks, including significant robustness in missing-modality scenarios and high efficiency under model compression.

4. Experimental Results

The paper evaluates OneTrackerV2 against SOTA trackers across 12 benchmarks:

RGB Tracking: OneTrackerV2 outperforms existing RGB-only trackers (e.g., ODTrack, ARTrackV2) on benchmarks like LaSOT, TrackingNet, and GOT-10k, despite being trained on a unified multimodal dataset. This proves the unified design does not compromise single-modality performance.
RGB+X Tracking: On Depth, Thermal, Event, and Language tracking benchmarks (e.g., DepthTrack, RGBT234, VisEvent, TNL2K), OneTrackerV2 surpasses methods that rely on task-specific architectures or fine-tuning (e.g., SUTrack, ViPT, STTrack).
Robustness to Missing Modalities: In scenarios where auxiliary modalities are absent (e.g., DepthTrack $_{miss}$ ), OneTrackerV2 significantly outperforms prior unified methods (e.g., SUTrack, SeqTrackv2), validating the effectiveness of the Meta Merger and stochastic training.
Efficiency and Compression: The model maintains high inference speeds (up to 72.4 FPS for the B224 variant). When compressed to a 6-layer variant, it retains strong performance (only ~2% drop) while achieving 159 FPS, outperforming compressed baselines like SuTrack-T224 in both speed and accuracy.

5. Significance and Claims

The authors claim that OneTrackerV2 represents a paradigm shift in multimodal tracking by demonstrating that scalability, robustness, and generalization can be achieved without sacrificing efficiency or accuracy.

Unified Training: The paper argues that a single-step training process with a shared architecture is superior to multi-stage fine-tuning, which often leads to suboptimal convergence.
Decoupled Learning: By explicitly separating motion modeling (T-MoE) from modality fusion (M-MoE), the framework resolves the "feature conflict" inherent in naive unified models, allowing the model to learn distinct, complementary features.
Practical Robustness: The framework is highlighted as particularly significant for real-world applications where sensor availability may vary (e.g., a thermal camera failing), as it maintains performance without requiring architectural changes or retraining.

The authors conclude that OneTrackerV2 sets a new standard for unified multimodal tracking, offering a scalable solution that balances high accuracy, inference efficiency, and resilience to input variations.