💻 computer science

Multi-Modal Guided Multi-Source Domain Adaptation for Object Detection

This paper proposes MS-DePro, a novel multi-source domain adaptation framework for object detection that leverages depth maps and text prompts to generate domain-agnostic region proposals and align learnable text embeddings, thereby achieving state-of-the-art performance on MSDA benchmarks.

Original authors: Sangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sangin Lee, Seokjun Kwon, Jeongmin Shin, Namil Kim, Yukyung Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a security guard to spot specific people in a crowd.

The Problem: The "One-Size-Fits-All" Trap
Usually, you train this guard using photos taken in bright daylight (Source Domain 1) and maybe some photos taken at night (Source Domain 2). You want the guard to work perfectly in a new, unseen environment, like a foggy morning or a rainy street (Target Domain).

The problem is that if you just mix the day and night photos together and say, "Learn from all of these," the guard gets confused. The day photos have bright colors and sharp shadows; the night photos are dark and grainy. The guard tries to find a "middle ground" that doesn't really exist, or worse, it gets distracted by the specific lighting of the training photos and fails when the weather changes. This is called the Domain Shift problem.

The Old Solution: Trying to "Forget" the Details
Previous methods tried to solve this by forcing the guard to ignore the specific details (like the color of the sky or the texture of the road) and focus only on the "shape" of the people. They did this by playing a game where the guard had to pretend it didn't know which photo came from which source. However, the paper argues this is like trying to learn the shape of a car by staring at a blurry, washed-out photo. It's hard to separate the "shape" from the "lighting" when you only have one type of picture (RGB/standard photos).

The New Solution: MS-DePro (The "Depth and Text" Detective)
The authors propose a new system called MS-DePro. Instead of just looking at standard photos, they give the guard two extra tools to help it understand the world better, regardless of the weather or lighting:

The "Depth Map" (The 3D Blueprint):
- The Analogy: Imagine looking at a photo of a car. In a photo, a red car and a blue car look very different. But if you look at a depth map (a black-and-white image showing how far away things are), both cars look like the same 3D shape. A tree is a tree, and a car is a car, regardless of whether it's day, night, or raining.
- How it helps: The system uses a special tool to generate these depth maps from the photos during training. It uses these maps to find where the objects are (localization). Because depth is about geometry and not color, it doesn't get confused by the sun or the rain. It tells the guard, "Hey, there's a solid object right here," even if the photo is dark.
The "Text Prompt" (The Smart Label):
- The Analogy: Imagine the guard has a notebook. Instead of just looking at the picture, the guard reads a label like "A person."
- How it helps: The system splits this label into two parts:
  - The Universal Part: "A person" is always a person, whether they are in a comic book or a real photo. The system uses the Depth Map to reinforce this universal truth.
  - The Specific Part: "A person in a rainy coat" is specific to the weather. The system uses the Photo to learn these specific details.
- By separating these two, the guard learns the core concept of "person" (which never changes) while still adapting to the specific look of the current scene.

How It Works in Practice
During the training phase (the "learning" phase), the system uses the photos, the generated depth maps, and the text labels all together. It's like the guard is studying with a 3D model and a dictionary.

However, once the training is done and the guard goes to work (the "inference" phase), the depth maps are no longer needed. The guard has already learned the 3D shapes and the universal concepts. It can now look at a new, foggy photo and say, "That's a car," because it learned the shape of a car from the depth maps and the concept of a car from the text, not just the color of the car in the training photos.

The Results
The paper claims that this method is the best so far (State-of-the-Art).

It beats other methods that try to mix different photo sources together.
It works better when moving from day to night, or from real photos to computer-generated images.
It even works well in "unseen" weather conditions (like heavy rain or fog) that it was never explicitly trained on, proving that it learned the true "shape" of objects rather than just memorizing the lighting.

In a Nutshell
Instead of trying to force a computer to ignore the differences between day and night photos, this new method gives the computer a 3D blueprint (depth) and a smart description (text) to understand what objects really are. This allows the computer to recognize objects in any weather or lighting condition, just like a human who understands that a car is a car, whether it's sunny, rainy, or dark.

Technical Summary: Multi-Modal Guided Multi-Source Domain Adaptation for Object Detection

1. Problem Statement

General object detection (OD) models often fail when deployed in target domains that differ from their training source domains due to domain shifts caused by factors such as time of day, style (real vs. synthetic), and camera artifacts. While Unsupervised Domain Adaptation (UDA) methods have addressed single-source scenarios, they struggle when multiple source domains are available. Naively blending multiple sources for single-source UDA often underperforms even a single source due to discrepancies among the sources.

Existing Multi-Source Domain Adaptation (MSDA) methods attempt to mitigate this by treating source domains separately, typically employing a domain-agnostic network (using Gradient Reversal Layers) and a domain-specific network. However, these methods suffer from a fundamental limitation: they attempt to learn domain-agnostic features solely from domain-specific RGB images by suppressing domain-specific cues. This creates a training conflict where the model must simultaneously suppress and encode domain-specific information from the same input, leading to optimization difficulties and slow convergence. Furthermore, RGB features are inherently sensitive to variations in illumination, color, and texture, making them a constrained source for true domain-agnostic representation.

2. Methodology: MS-DePro

To address these limitations, the authors propose MS-DePro (Multi-Source Detector with Depth and Prompt). The core philosophy is to explicitly disentangle domain-specific and domain-agnostic features by leveraging additional, inherently invariant input modalities: depth maps and text.

The framework follows a Mean-Teacher architecture, utilizing a labeled multi-source student and an unlabeled target teacher. It comprises two primary components:

A. Depth-Guided Localization

Recognizing that depth maps provide consistent representations of object shape and structure regardless of lighting or texture, the authors use them to guide object localization.

Mechanism: During training, an off-the-shelf monocular depth estimation model generates relative depth maps from input RGB images.
Architecture: The system employs decoupled backbones and Region Proposal Networks (RPNs) for both RGB ( $BB_R, RPN_R$ ) and Depth ( $BB_D, RPN_D$ ).
Integration: Region proposals from the depth branch ( $p_D$ ) are projected onto the RGB feature map. The system aggregates proposals from both modalities, selecting the top- $k$ based on objectness logits. This allows the model to capture regions missed by RGB alone and filter out spurious proposals.
Inference: The depth branch is used exclusively during training to generate pseudo-labels and refine the student's localization. At inference, the depth branch is removed, and detection is performed using only the RGB network, incurring no additional latency.

B. Multi-Modal Guided Prompt Learning

The authors leverage the high-level semantic consistency of text to encode domain-agnostic knowledge while using RGB features for domain-specific adaptation.

Prompt Decomposition: The learnable text prompt is decomposed into two sets of tokens:
1. Domain-Agnostic Tokens ( $v_m$ ): Enriched with features from shallow depth maps ( $\hat{F}_D$ ) via a Domain-Agnostic Network (DA-Net). These capture shared geometric and structural characteristics across domains.
2. Domain-Specific Tokens ( $d_m$ ): Enriched with features from shallow RGB maps ( $\hat{F}_R$ ) via a Domain-Specific Network (DS-Net). These capture style and appearance cues specific to each domain.
Token Buffers: Since depth maps are unavailable at inference, the Domain-Agnostic token ( $\pi_{DA}$ ) and the Domain-Specific token ( $\pi_{DS}$ ) are pre-computed and stored in buffers during training using Exponential Moving Average (EMA). These buffers replace the dynamic token generation during inference, allowing the model to use a static, optimized text prompt without computational overhead.
Classification: The final text prompt combines these tokens with class names (e.g., [v1]...[vM1][d1]...[dM2][CLS]) to align with object embeddings for classification.

3. Key Contributions

Novel Framework: Introduction of MS-DePro, the first MSDA framework for object detection that explicitly leverages auxiliary modalities (depth and text) to encode domain-invariant representations, bypassing the need for adversarial learning to suppress domain-specific cues.
Depth-Guided Localization: A method that utilizes depth maps to generate complementary region proposals, improving localization robustness against domain shifts in texture and illumination.
Multi-Modal Guided Prompt Learning: A structured prompt learning approach that explicitly separates domain-agnostic (depth-guided) and domain-specific (RGB-guided) knowledge within learnable text embeddings.
State-of-the-Art Performance: Comprehensive experiments demonstrating that MS-DePro achieves superior performance on MSDA benchmarks compared to existing methods.

4. Experimental Results

The authors evaluated MS-DePro on standard MSDA benchmarks and extended the evaluation to Multi-Source Domain Generalization (MSDG).

MSDA Benchmarks:
- Cross-Time Adaptation (BDD100K): MS-DePro achieved 53.7 mAP, outperforming the previous SOTA (ACIA) by 5.8 mAP. Notably, it surpassed the "Oracle All-combined" baseline (trained on both source and labeled target data) by 2.9 mAP.
- Cross-Camera Adaptation (Cityscapes/KITTI): MS-DePro achieved 68.4 mAP, a 9.3 mAP improvement over ACIA.
- Mixed-Domain Adaptation: In settings combining Cityscapes, MS-COCO, and Synscapes, MS-DePro outperformed the previous SOTA by 2.9 to 4.4 mAP depending on the number of source domains.
Domain Generalization (MSDG):
- Diverse Weather: MS-DePro achieved the highest average mAP across unseen weather domains (mPC), outperforming methods like PDOC and CLIP-Gap.
- Real to Artistic: The model demonstrated strong generalization from real-world datasets to artistic styles (Clipart, Comic, Watercolor), significantly outperforming single-source domain generalization baselines.
Ablation Studies:
- Depth vs. RGB: t-SNE visualizations confirmed that depth features form a single, consistent cluster across domains, whereas RGB features are scattered.
- Component Synergy: Removing either depth-guided localization or multi-modal prompt learning resulted in significant performance drops, confirming their synergistic effect.
- Token Configuration: The best performance was achieved by using image-level features for domain-agnostic tokens and proposal-level features for domain-specific tokens.
- Computational Cost: While training involves additional depth processing, inference cost is identical to the baseline RGB-only model, as the depth branch is removed and tokens are pre-computed.

5. Significance and Limitations

The paper claims that MS-DePro demonstrates the effectiveness of explicitly disentangling domain-specific and domain-agnostic features using auxiliary modalities. By moving away from the "paradoxical" process of suppressing domain cues from RGB images, the method achieves more robust adaptation. The results suggest that leveraging geometric cues (depth) and semantic cues (text) at the input level is a superior strategy for MSDA compared to relying solely on adversarial alignment.

Limitations:
The authors acknowledge that depth maps generated by monocular estimation models can be noisy under challenging conditions such as motion blur, severe occlusion, adverse weather, and low light. In these scenarios, the quality of region proposals and the stability of the Domain-Agnostic token may degrade, potentially affecting performance. The paper suggests that improving depth estimation robustness is a direction for future work.