🤖 AI

DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

This paper introduces DRMOT, a novel RGBD Referring Multi-Object Tracking task, along with the DRSet dataset and the DRTrack framework, to address the limitations of 2D-only models by leveraging fused RGB, depth, and language modalities for robust 3D-aware target tracking and spatial-semantic grounding.

Original authors: Sijia Chen, Lijuan Ma, Yanqiu Yu, En Yu, Liman Liu, Wenbing Tao

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sijia Chen, Lijuan Ma, Yanqiu Yu, En Yu, Liman Liu, Wenbing Tao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific friend in a crowded, busy room. You ask your smart assistant, "Find the person closest to the camera."

If your assistant only has a 2D photograph (like a flat picture on a phone), it gets confused. In a flat picture, everyone looks like they are on the same flat plane. The person standing right next to the lens and the person standing far back might look the same size if they are wearing similar clothes. The assistant might guess wrong, pointing at the person in the back instead of the one in front. This is the problem with current tracking technology: it sees the world like a flat painting, not a real room.

This paper introduces a new way to solve this by giving the assistant 3D glasses (depth information) and a super-brain (a large language model).

Here is a simple breakdown of what the authors did:

1. The New Game: DRMOT

The authors created a new challenge called DRMOT (Depth Referring Multi-Object Tracking).

The Old Way: You give the computer a video and a sentence like "Track the guy in the red hat." The computer tries to follow him using only the video colors.
The New Way: You give the computer the video, the sentence, AND a "depth map" (a special image that tells the computer exactly how far away every pixel is). Now, if you say "Track the person closest to the camera," the computer can actually measure the distance and pick the right person, even if they look identical to someone far away.

2. The New Map: DRSet

To teach computers this new skill, the authors built a special training library called DRSet.

Think of this as a massive library of 187 different "scenes" (like a park, a living room, or a street).
For every scene, they have the regular video, the 3D depth map, and 240 specific instructions written in human language.
Crucially, 56 of those instructions rely on distance (e.g., "the car behind the tree" or "the person nearest to us"). This forces the computer to learn how to use 3D space to understand language.

3. The New Brain: DRTrack

They also built a new system called DRTrack to solve these puzzles. It works in two steps, like a detective with two tools:

Step 1: The "Eagle Eye" (The MLLM):
They use a powerful AI brain (a Multimodal Large Language Model) that looks at the video, the depth map, and the sentence all at once. It's like a detective who can see the room in 3D. When you say, "Find the closest person," this brain uses the depth map to instantly know who is actually in front and who is in the back. It draws a box around the correct person.
- Analogy: It's like having a laser pointer that measures distance while you read a map.
Step 2: The "Sticky Tape" (The Tracker):
Once the brain finds the person in the first frame, the system needs to keep following them as they move, even if they get blocked by a wall or a crowd. The authors added "depth tape" to their tracking method.
- Usually, if two people walk close together, the computer might mix them up (swapping their IDs).
- With DRTrack, the computer checks the 3D distance. If Person A is 2 meters away and Person B is 5 meters away, the computer knows they are different people, even if they look very similar. This keeps their identities separate and prevents confusion.

4. The Results

The authors tested their new system against old systems that only used flat 2D videos.

The Outcome: The new system (DRTrack) was much better at finding the right person and keeping track of them without getting confused.
Why it matters: It proved that giving AI "depth vision" is the key to understanding instructions that involve space and distance, which is something flat 2D cameras simply cannot do on their own.

In short: The paper says, "If you want an AI to understand instructions like 'the one closest to us,' you can't just show it a flat video. You have to show it the 3D depth of the room, too. We built a new dataset and a new AI brain to prove this works."

Technical Summary: DRMOT: A Dataset and Framework for RGBD Referring Multi-Object Tracking

1. Problem Statement

Referring Multi-Object Tracking (RMOT) aims to identify and track specific targets based on natural language descriptions, a capability critical for interactive AI systems like robotics and autonomous driving. However, existing RMOT approaches rely exclusively on 2D RGB data. This limitation creates significant challenges when language descriptions involve complex spatial semantics dependent on depth (e.g., "the person closest to the camera") or when targets suffer from severe occlusion. Without explicit 3D spatial information, 2D models struggle to resolve spatial hierarchies, leading to ambiguous reasoning, incorrect target grounding, and identity switches.

To address these limitations, the authors propose a novel task: RGBD Referring Multi-Object Tracking (DRMOT). This task requires models to fuse RGB, Depth (D), and Language (L) modalities to achieve 3D-aware tracking, effectively grounding semantic descriptions in a geometric context while maintaining temporal identity consistency.

2. Methodology

The paper introduces two primary components to advance the DRMOT task: a tailored dataset (DRSet) and a tracking framework (DRTrack).

2.1. The DRSet Dataset

DRSet is constructed to evaluate spatial-semantic grounding and tracking capabilities under real-world 3D conditions.

Composition: The dataset comprises 187 diverse scenes (indoor, outdoor, night, complex weather) containing synchronized RGB images and depth maps.
Annotations: It includes 240 language descriptions. Notably, 56 of these descriptions explicitly incorporate depth-related information (e.g., relative distance, front/back positioning).
Structure: The dataset is split into 141 training videos and 99 testing videos at the scene level to prevent data leakage. It covers 18 object categories (humans, animals, vehicles, objects) with varying counts per sequence and frame lengths ranging from short to over 1,000 frames.
Annotation Process: A rigorous four-step process was employed: (1) creating an attribute table for static and dynamic features; (2) selecting representative targets; (3) frame-by-frame bounding box annotation and language composition; and (4) a two-person cross-review protocol to ensure coordinate validity and semantic consistency.

2.2. The DRTrack Framework

DRTrack is a Multimodal Large Language Model (MLLM)-guided framework designed for robust grounding and stable cross-frame association. It operates in two stages:

Stage 1: Depth-Promoted Language Grounding

Input: The framework utilizes an MLLM (specifically Qwen2.5-VL-3B) that concurrently processes three modalities: Language descriptions ( $L$ ), RGB images ($RGB$), and Depth maps ( $D$ ). Depth maps are converted into 3-channel pseudo-RGB representations.
Objective: The MLLM is tasked with outputting precise 2D bounding boxes for targets described in the language, leveraging depth cues to resolve 2D ambiguities.
Training Strategy: The MLLM is fine-tuned using Group Relative Policy Optimization (GRPO). A specialized reward function ( $R_{total} = R_{format} + R_{IoU}$ $R_{t o t a l} = R_{f or ma t} + R_{I o U}$ ) is employed:
- Format Reward: Ensures structured JSON output for reliable box extraction.
- IoU Reward: Encourages geometric consistency between predicted and ground-truth boxes using the Hungarian algorithm for matching.

Stage 2: Depth-Enhanced OC-SORT Association

Mechanism: Building on the observation-centric OC-SORT tracker, this stage integrates depth information to refine trajectory association.
RGBD Joint Similarity ( $S_{RGBD}$ ): A new similarity metric is defined to combine 2D appearance and 3D depth consistency:
$S_{RGBD} = \alpha \cdot \text{IoU} + (1 - \alpha) \cdot S_D$
Where $S_D$ is an exponential decay function of the absolute depth difference between the detection and the track prediction.
Association Cost: The final assignment cost minimizes a combination of the RGBD joint similarity and a Velocity Direction Consistency (VDC) motion prior:
$C = -(S_{RGBD} + \lambda \cdot \text{VDC})$
This dual-constrained cost function mitigates false associations caused by 2D occlusion or proximity.

3. Key Contributions

DRMOT Task Definition: The formalization of RGBD Referring Multi-Object Tracking, which explicitly integrates RGB, Depth, and Language modalities to handle 3D-aware spatial reasoning and tracking.
DRSet Dataset: The construction of a comprehensive dataset featuring 187 scenes and 240 descriptions (including 56 depth-specific), providing a benchmark for evaluating spatial-semantic grounding in 3D environments.
DRTrack Framework: A novel MLLM-guided architecture that achieves state-of-the-art performance by combining geometric-aware GRPO fine-tuning for grounding with depth-enhanced OC-SORT for robust association.

4. Experimental Results

Extensive experiments were conducted on the DRSet test set, evaluating performance using Higher Order Tracking Accuracy (HOTA) and its components (Detection Accuracy, Association Accuracy, etc.).

Performance: DRTrack achieved a HOTA of 33.24%, significantly outperforming all baselines.
- It surpassed the strongest zero-shot MLLM baseline (Qwen2.5-VL-3B with L+RGB input, HOTA: 15.13%) by over 18 percentage points.
- It also outperformed previous RMOT methods (TransRMOT, TempRMOT) and other MLLMs (DeepSeek-VL, LLaVA series) that relied solely on RGB inputs.
Ablation Studies:
- Depth Modality: Adding depth information to the MLLM input (L+RGB+D) without fine-tuning increased HOTA from 15.13% to 32.68%, demonstrating the critical value of geometric cues.
- GRPO Fine-tuning: Applying geometric-aware GRPO fine-tuning further improved HOTA from 32.68% to 33.24%, confirming the efficacy of the reward-based optimization.
- Fusion Weight ( $\alpha$ ): Sensitivity analysis showed that a fusion weight of $\alpha=0.9$ (prioritizing 2D IoU while incorporating 10% depth constraint) yielded optimal results, balancing spatial context with geometric robustness.

5. Significance and Claims

The paper claims that the integration of depth information is essential for disambiguating referring expressions that are indistinguishable in 2D image space. By proposing DRMOT and the DRSet dataset, the authors provide a necessary resource for evaluating models on spatial-semantic grounding under geometric ambiguity.

The DRTrack framework demonstrates that synergizing depth cues with MLLMs can effectively resolve spatial ambiguities (e.g., distinguishing "closest" vs. "farthest" objects) and maintain identity consistency under severe occlusion. The authors position DRMOT as a foundational step toward more robust, 3D-aware interactive AI systems, offering a strong baseline for future research in this emerging domain. The work is presented as an advancement in machine learning perception, with no specific societal consequences highlighted beyond the general goal of improving environmental understanding for autonomous systems.