Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

1. Problem Statement

Camouflaged Object Detection (COD) aims to segment objects that blend seamlessly into complex backgrounds, a task critical for applications in medical imaging, industrial inspection, and remote sensing. While RGB-only methods have advanced, they often fail in scenarios with strong foreground-background similarity and weak boundary contrast.
Multi-modal COD (integrating RGB with Depth, Thermal, or Polarization) has emerged to improve robustness. However, existing multi-modal approaches suffer from two main limitations:

• Modality-Specific Architectures: Most methods require custom-designed fusion strategies or dual-stream architectures tailored to specific modality pairs (e.g., RGB-Depth vs. RGB-Thermal).
• Poor Scalability: Adapting these models to new auxiliary modalities requires significant architectural redesign, limiting cross-modal generalization and scalability.
• SAM Limitations: While the Segment Anything Model (SAM) offers strong zero-shot capabilities, standard fine-tuning or adapter-based approaches for COD remain modality-specific and struggle to generalize across different auxiliary inputs without structural changes.

2. Methodology

The authors propose a Modality-Agnostic Prompt Learning Framework that adapts the frozen Segment Anything Model (SAM) to arbitrary auxiliary modalities (Depth, Thermal, Polarization) without redesigning the core architecture. The framework consists of three main components:

A. Dual-Domain Learning Paradigm

Instead of fusing raw features in a complex manner, the method models multi-modal learning through the interaction of two distinct domains:

1. Content Domain (Data-Driven):
   • Constructed by element-wise summation of RGB features (extracted from a frozen SAM image encoder) and auxiliary modality features (extracted via a frozen PVT backbone).
   • This domain aggregates heterogeneous spatial cues and semantic information (e.g., 3D geometry from depth, thermal contrast from infrared, material properties from polarization).
2. Prompt Domain (Knowledge-Driven):
   • Initialized with a set of learnable prompt prototypes that act as a global knowledge base for camouflaged objects.
   • These prototypes are adaptively modulated by the Content Domain features via a lightweight attention layer, generating adaptive prompts that encode task-relevant guidance signals.

B. Content-Prompt Interaction Module (CPIM)

To bridge the gap between data-driven evidence and knowledge-driven priors, the framework employs a symmetric cross-attention mechanism:

• Bidirectional Synergy: The module performs cross-attention between the Content Domain and the Prompt Domain. This allows the model to distill task-relevant evidence from heterogeneous observations into the prompt prototypes while simultaneously rectifying spatial fidelity using structured priors.
• Auxi-Tokens: A set of learnable "Auxi-tokens" is introduced and concatenated with SAM's original tokens to further enhance the network's flexibility and modality-agnostic nature.

C. Mask Refine Module (MRM)

Recognizing that SAM's initial predictions can be coarse in low-contrast environments:

• The MRM takes the refined prompt domain features and modulates the internal representations of the SAM mask decoder (specifically the Token-to-Image Attention and Transposed Convolution features).
• This process reconstructs the internal features to sharpen boundaries and calibrate coarse predictions, resulting in precise object segmentation.

3. Key Contributions

1. Unified Modality-Agnostic Framework: The first framework to seamlessly integrate RGB with arbitrary visual modalities (Depth, Thermal, Polarization) for COD using a single, unified architecture, eliminating the need for modality-specific designs.
2. Dual-Domain Learning Paradigm: Introduction of a novel learning strategy that decouples raw perceptual data (Content Domain) from task priors (Prompt Domain), utilizing bidirectional cross-attention to synergize data-driven and knowledge-driven representations.
3. Parameter Efficiency: The approach achieves state-of-the-art performance with a minimal number of trainable parameters (only 2.73M), significantly outperforming existing SAM-based fine-tuning methods that often require tens or hundreds of millions of parameters.
4. Robust Generalization: Demonstrated strong cross-modal transfer capabilities, where models trained on one modality combination (e.g., RGB-Depth) maintain high performance when tested on unseen modalities (e.g., RGB-Thermal).

4. Experimental Results

The framework was evaluated on three major multi-modal COD benchmarks:

• RGB-Depth (COD10K, CAMO, NC4K, CHAMELEON): Achieved State-of-the-Art (SOTA) performance, outperforming the runner-up by margins of 0.002 (Sα), 0.013 (Fβ), and 0.008 (Eϕ) on COD10K.
• RGB-Polarization (PCOD-1200): Achieved SOTA across all metrics (Sα: 0.945, Fβ: 0.924), effectively leveraging polarization cues for texture differentiation.
• RGB-Thermal (VIAC): Demonstrated superior performance across diverse camouflage types (plant, nets, suits), validating generalization in thermal scenarios.

Efficiency Comparison:

• The proposed method uses only 2.73M trainable parameters, compared to 5.1M–121.9M for other SAM-based methods (e.g., DSAM, MM-SAM), while delivering higher accuracy.
• Cross-Modality Transfer: When trained on RGB+Depth and tested on RGB+Polarization or RGB+Thermal, the performance drop was minimal (<0.021 in Sα), confirming the modality-agnostic nature of the design.

Generalizability:

• The method was also extended to Salient Object Detection (SOD) on RGB-Thermal datasets, outperforming existing SOTA methods, proving its versatility beyond COD.

5. Significance

This paper addresses a critical bottleneck in multi-modal computer vision: the lack of scalable, generalizable architectures for integrating diverse sensor data. By shifting the focus from modality-specific fusion to modality-agnostic prompt learning, the authors provide a blueprint for efficient adaptation of large foundation models (like SAM) to complex, multi-sensor tasks. The work demonstrates that high-performance segmentation can be achieved with minimal parameter overhead, making it highly suitable for deployment in resource-constrained environments and adaptable to future sensor modalities without architectural re-engineering.