💻 computer science

Revisiting RGBT Tracking Benchmarks from the Perspective of Modality Validity: A New Benchmark, Problem, and Solution

This paper introduces MV-RGBT, a new benchmark capturing RGBT tracking scenarios with invalid modalities to highlight the limitations of existing datasets, proposes the "when to fuse" problem to guide fusion strategies in adverse conditions, and presents MoETrack, a mixture-of-experts solution that achieves state-of-the-art performance by demonstrating that fusion is not always beneficial.

Original authors: Zhangyong Tang, Tianyang Xu, Zhenhua Feng, Xuefeng Zhu, Chunyang Cheng, Xiao-Jun Wu, Josef Kittler

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhangyong Tang, Tianyang Xu, Zhenhua Feng, Xuefeng Zhu, Chunyang Cheng, Xiao-Jun Wu, Josef Kittler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why Do We Need Two Eyes?

Imagine you are trying to find a lost friend in a crowded park.

The "RGB" Eye (Visible Light): This is like your normal vision. It sees colors and details beautifully when the sun is shining. But if it gets dark, foggy, or if your friend hides behind a glass window, your normal vision fails.
The "TIR" Eye (Thermal Infrared): This is like a heat-sensing night-vision goggles. It doesn't care about the dark or fog; it sees body heat. But it can't see colors, and if your friend is standing next to a hot radiator or a mirror, the heat signature gets confusing.

RGBT Tracking is the art of using both eyes at the same time to find that friend. The idea is that if one eye goes blind, the other can take over, making the search much more reliable.

The Problem: The "Perfect Weather" Trap

The authors of this paper noticed a huge flaw in how scientists were testing these "two-eye" trackers.

The Old Benchmarks (The Test Tracks): Imagine a driving school testing a self-driving car. They only built test tracks on sunny, clear days with perfect roads.
The Reality: In the real world, cars have to drive in rain, snow, and fog.
The Issue: The existing datasets for RGBT tracking were like those perfect driving tracks. They mostly contained videos taken in good weather where both the normal camera and the heat camera worked perfectly. Because of this, the trackers learned to just "mix" the two images together all the time, assuming both were helpful.

The authors realized: "We are training our trackers for a world that doesn't exist. We need to test them in the storm."

The Solution: A New "Stormy" Test Track (MV-RGBT)

To fix this, the team created a new benchmark called MV-RGBT.

The Analogy: Instead of a sunny park, they filmed videos in Multi-Modality Warranting (MMW) scenarios. These are the "stormy days" of tracking:
- Nighttime: The normal camera sees nothing (black), but the heat camera sees the person clearly.
- Heavy Fog: The normal camera is blurry, but the heat camera cuts through.
- Glass/Mirrors: The heat camera gets confused by reflections, but the normal camera sees the person clearly.
- Overexposure: The sun is too bright for the normal camera, but the heat camera is fine.

In this new dataset, they proved that often, one of the cameras is useless. Sometimes the "RGB" eye is blind, and sometimes the "TIR" eye is blind.

The New Question: "When Should We Mix the Ingredients?"

Because one camera is often useless in these bad conditions, the authors asked a new question: "When should we actually combine the data?"

The Old Way: Always mix the two camera feeds together, like making a smoothie with two fruits, even if one of them is rotten. If you blend a rotten fruit into a smoothie, the whole drink tastes bad.
The New Insight: In bad weather, you shouldn't always mix. Sometimes you should just trust the one camera that is working and ignore the broken one.

The New Method: The "Expert Panel" (MoETrack)

To solve the "When to mix?" problem, they built a new tracker called MoETrack (Mixture of Experts).

Imagine a Panel of Three Judges deciding the winner of a contest:

Judge RGB: Only looks at the color camera.
Judge TIR: Only looks at the heat camera.
Judge Fused: Looks at both cameras mixed together.

How it works:

In every single frame of the video, these three judges give their opinion and a Confidence Score (a number saying, "How sure am I?").
The system acts like a smart referee. It looks at the scores.
- If the video is foggy, Judge RGB will say, "I can't see anything!" (Low score). Judge TIR will say, "I see him clearly!" (High score). The system picks Judge TIR's answer.
- If the weather is perfect, Judge Fused might say, "Combining both gives me the best view!" (Highest score). The system picks the mixed answer.

The Metaphor: It's like having a team of specialists. If you are in a dark room, you don't ask the color-blind specialist to lead; you ask the night-vision specialist. If you are in a bright room, you ask the color specialist. You don't force them to work together if one of them is useless.

The Results: Why This Matters

The authors tested their new "Expert Panel" method on:

Their new "Stormy" dataset (MV-RGBT).
The old "Sunny" datasets (like RGBT234, LasHeR).

The Findings:

On the Stormy Days: The old methods (which always mix) failed miserably because they were trying to mix bad data. The new MoETrack method won easily because it knew when to ignore the broken camera.
On the Sunny Days: Even on perfect days, the new method was still the best. It turned out that being smart about when to mix is better than just always mixing.

Summary

This paper is a wake-up call for the computer vision world.

Stop testing in perfect conditions. We need to test AI in the "bad weather" where it actually needs to work.
Don't always mix. Just because you have two sources of information doesn't mean you should combine them. Sometimes, the best strategy is to trust the one that works and ignore the one that doesn't.
The "Expert Panel" approach is the future. It allows AI to be flexible, switching between strategies depending on what the environment looks like, leading to much more reliable tracking in the real world.

1. Problem Statement

The paper identifies a critical gap between existing RGBT (RGB + Thermal Infrared) tracking benchmarks and real-world application scenarios, specifically Multi-Modality Warranting (MMW) scenarios.

The Issue: Current benchmarks (e.g., RGBT234, LasHeR) are predominantly collected in "common scenarios" where both RGB and TIR modalities provide high-quality, complementary information. They fail to represent severe imaging conditions (nighttime, heavy fog, over-exposure, TIR reflection/truncation) where one modality becomes non-informative or invalid.
The Consequence: Existing tracking methods, which typically fuse data at every frame, are untested in MMW scenarios. In these conditions, indiscriminate fusion can degrade performance because injecting "noisy" or invalid data from one modality can corrupt the reliable data from the other.
The Core Question: The authors pose a new problem: "When to fuse?" rather than just "How to fuse?" They argue that fusion is not always beneficial, especially when one modality is invalid.

2. Methodology: MoETrack

To address the "when to fuse" problem, the authors propose MoETrack, a novel tracking framework based on a Mixture of Experts (MoE) architecture.

Architecture:
- Shared Backbone: Utilizes a Vision Transformer (ViT-base) backbone shared across RGB, TIR, and fused branches.
- Three Experts: The network employs three parallel tracking heads (experts):
  1. RGB Expert: Processes only RGB input.
  2. TIR Expert: Processes only TIR input.
  3. RGBT Expert: Processes fused RGB+TIR input (via element-wise addition).
- Decision-Level Fusion: Instead of fusing features at the pixel or feature level for every frame, the method operates at the decision level.
Selection Mechanism:
- During inference, each expert generates a bounding box prediction and a confidence score (derived from the maximum value of the classification map).
- The final prediction is selected based on the highest confidence score ( $m_c = \max(c_{sRGB}, c_{sTIR}, c_{sRGBT})$ ).
- Logic: If the RGBT expert has the highest score, fusion is deemed beneficial. If the RGB or TIR expert scores higher, the system relies on the single valid modality, effectively "skipping" fusion when it is counterproductive.
Training: The backbone is fine-tuned, while the expert heads are trained from scratch. The total loss is the average of the individual losses for the three experts, encouraging specialization.

3. Key Contributions

A. New Benchmark: MV-RGBT

The authors introduce MV-RGBT, a new dataset specifically captured in MMW scenarios to bridge the gap between research and reality.

Composition: Contains 122 video pairs (89.9k frames) with 36 object categories across 19 diverse scenes.
Unique Feature: Unlike other benchmarks, 100% of the data is collected in challenging conditions (fog, night, glare, TIR reflection).
Stratification: The dataset is divided into two subsets based on modality validity:
- MV-RGBT-RGB: Scenarios where RGB is valid, but TIR is challenged.
- MV-RGBT-TIR: Scenarios where TIR is valid, but RGB is invalid (e.g., total darkness).
Significance: It forces researchers to evaluate modality validity and robustness rather than just fusion capability.

B. New Problem Formulation

The paper formally defines the "When to Fuse" problem. It challenges the assumption that multi-modality fusion is universally superior, demonstrating that in MMW scenarios, fusing invalid data can be detrimental.

C. New Method (MoETrack)

MoETrack provides a flexible baseline that adaptively switches between single-modality and fused tracking based on real-time confidence, offering a solution to the "when to fuse" problem without requiring complex data validity classifiers.

4. Experimental Results

The authors conducted extensive experiments on MV-RGBT and standard benchmarks (GTOT, RGBT234, LasHeR, VTUAV-ST).

State-of-the-Art (SOTA) Performance:
- MV-RGBT: MoETrack achieved 51.4% PR and 67.6% SR, outperforming the second-best tracker (GMMT) by 2.3% and 2.2% respectively.
- Standard Benchmarks: MoETrack set new SOTA records on RGBT234 (65.1% SR), LasHeR (72.1% PR, 57.8% SR), and VTUAV-ST (83.6% PR, 69.5% SR).
Ablation Studies:
- Comparing MoETrack against its single-expert variants (MoETrack-RGB, MoETrack-TIR) and a standard fused baseline (SETrack) confirmed that the adaptive selection mechanism is crucial for performance, particularly on MV-RGBT.
- On MV-RGBT, the single-expert performance varied drastically depending on the subset (e.g., TIR expert failed on RGB-valid subsets), proving that a fixed fusion strategy is insufficient.
Efficiency: MoETrack runs at 23 FPS, which is competitive with other SOTA methods (e.g., ViPT at 39 FPS, GMMT at 20 FPS) while offering superior accuracy.

5. Significance and Conclusion

Paradigm Shift: The paper shifts the focus from "how to better fuse" to "when to fuse." It demonstrates that indiscriminate fusion is harmful in MMW scenarios where one modality is invalid.
Benchmark Impact: MV-RGBT provides a rigorous testbed for evaluating the robustness of trackers in real-world, degraded environments, moving beyond the "clean" data of previous benchmarks.
Generalizability: The insights regarding modality validity and the MoE strategy are applicable beyond RGBT tracking to other multi-modal tasks (e.g., RGB-D, RGB-E) and detection systems.
Practical Deployment: The findings suggest that for practical applications (nighttime surveillance, autonomous driving in fog), systems must dynamically assess modality quality rather than relying on static fusion rules.

In summary, this work redefines the landscape of RGBT tracking by introducing a challenging, realistic benchmark and a method that intelligently adapts to the validity of input modalities, proving that less fusion (or no fusion) can sometimes be better than more fusion.