💻 computer science

ECFNet: Enhanced Cross-modal Fusion Network for RGB-D Salient Object Detection

This paper proposes ECFNet, a Swin Transformer-based framework for RGB-D salient object detection that enhances performance by preserving modality-specific features through collaborative interaction mechanisms, including Cross-Gated Residual Fusion, Prediction-Guided Progressive Aggregation, Gated Skip Connections, and edge refinement modules, achieving state-of-the-art results across eight benchmark datasets.

Original authors: Mengjun Song, Jinggong Wei

Published 2026-09-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mengjun Song, Jinggong Wei

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, machines are constantly learning to see the world as humans do, but with a specific goal: to instantly spot the most important object in a cluttered scene. This task, known as salient object detection, is the digital equivalent of a person glancing at a busy street and immediately noticing a bright red balloon rather than the gray pavement or the passing cars. For years, computers have relied on standard color photographs to do this. However, just as a human might struggle to judge the distance of a foggy object or the shape of a dark silhouette using only a flat image, computers often get confused when the background looks too much like the foreground or when lighting is poor. To solve this, researchers have begun giving computers a second pair of eyes: depth maps. These are special images that record how far away every point in a scene is, providing a three-dimensional skeleton that complements the flat color picture. By combining these two views, machines can better understand the structure of the world, separating a nearby cup from a distant wall even if they share the same color.

Despite these advances, a significant hurdle remains. When computers try to merge the flat color picture with the 3D depth map, they often treat the two sources of information as if they were identical, forcing them to blend into a single, generic representation. This approach ignores the unique strengths of each view. The color image is excellent at showing texture and fine detail, while the depth map is superior at revealing shape and distance, but it is also prone to static and errors, much like a radio signal picking up interference. If a computer blindly mixes these two, the noise from the depth map can corrupt the clear details of the color image, leading to blurry or incorrect results. A new study by researchers at Tianjin University of Technology and Education addresses this specific problem by designing a system that respects the individual nature of each view while teaching them to work together more intelligently.

The researchers, Mengjun Song and Jinggong Wei, introduced a new framework called ECFNet, which stands for Enhanced Cross-modal Fusion Network. Instead of simply smashing the two types of data together, their system acts like a skilled editor who knows exactly when to listen to the color camera and when to trust the depth sensor. The core of their innovation is a method that allows the two streams of information to talk to each other without losing their own identity. They built a mechanism that acts as a gatekeeper, constantly checking the quality of the depth information. If the depth map is noisy or unreliable in a certain area, the system automatically dampens its influence, relying more heavily on the clear color details. Conversely, when the depth map offers strong structural clues, the system amplifies those signals to help define the edges of an object. This two-way conversation ensures that the final picture is not a muddy compromise, but a sharp, accurate representation that leverages the best of both worlds.

To handle objects of different sizes, from a tiny insect on a leaf to a large building in the distance, the system uses a progressive approach. It starts by looking at the big picture to understand the general layout of the scene, then gradually zooms in to refine the details. At each step of this zoom, the system uses its previous guess about where the object is to guide the next, more detailed look. This is similar to how a human might first spot a shape in the distance and then walk closer to confirm it is a person rather than a tree. By using this step-by-step guidance, the computer avoids getting distracted by irrelevant background clutter. Furthermore, the system includes a specialized module dedicated to sharpening the boundaries of the detected objects. This ensures that the final outline of the salient object is crisp and precise, rather than fuzzy or jagged, which is a common failure point in older methods.

The team tested their new system against seventeen other leading methods across eight different datasets, which included thousands of images ranging from indoor living rooms to outdoor landscapes. The results were striking. In more than half of the specific measurements used to judge performance, their method ranked in the top three, and it took the number one spot in eleven different categories. On one particularly difficult dataset known for its complex indoor scenes, the new system achieved the best possible scores on all four major metrics, outperforming the previous leaders. It was especially successful at handling challenging conditions, such as low-light environments where depth sensors often struggle, or scenes where the object and background have very similar colors. The system also proved to be efficient, capable of processing images at a speed of thirty-seven frames per second, which is fast enough for real-time applications like autonomous driving or robotics.

While the new system represents a significant leap forward, the researchers are careful to note that it is not perfect. They identified specific scenarios where the system still falters, such as when trying to detect extremely thin structures like a single vine or the delicate antennae of a butterfly, or when dealing with dense crowds where people overlap heavily. In these cases, the fine details can sometimes get lost, or the system might merge separate objects into one. However, the study clearly demonstrates that explicitly preserving the unique characteristics of each data source, rather than forcing them into a single mold, leads to a much more robust and accurate understanding of the visual world. By teaching the computer to listen to the right voice at the right time, the researchers have created a tool that sees the world with greater clarity and precision than ever before.

Technical Summary: ECFNet: Enhanced Cross-modal Fusion Network for RGB-D Salient Object Detection

Problem Statement
Salient Object Detection (SOD) aims to identify visually significant regions within complex scenes. While RGB-D SOD leverages the complementary nature of RGB appearance and depth structural information, existing methods face three primary limitations:

Insufficient Modality Preservation: Many approaches focus on learning shared representations, often failing to preserve the specific characteristics of individual modalities. This reduces fusion effectiveness, particularly when depth maps contain noise or significant modality differences exist.
Noise and Quality Disparities: Variations in quality between RGB and depth modalities, combined with frequent noise in depth maps, negatively impact fusion performance.
Architectural Constraints: While Convolutional Neural Networks (CNNs) excel at local feature extraction, they struggle with long-range dependencies. Conversely, while Transformers capture global context, effectively combining them with local refinement for cross-modal fusion remains challenging. Furthermore, existing fusion strategies often fail to adequately address multi-scale representation learning across different input resolutions.

Methodology: ECFNet Architecture
The authors propose ECFNet, a framework built upon the Swin Transformer backbone, designed to preserve modality-specific features through collaborative interaction. The network follows an encoder-decoder structure with four primary modules:

Cross-Gated Residual Fusion (CGRF):
- Located at the encoder stages, this module facilitates bidirectional interaction between RGB and depth features.
- It employs a bidirectional gating mechanism where channel attention signals from one modality guide the feature update of the other.
- A learnable residual parameter ( $\gamma$ ) balances the fused output with the identity mapping, allowing the network to start from an identity mapping and gradually learn cross-modal interactions.
- Depthwise separable convolutions are used to project features into a shared space, reducing computational cost.
Gated Skip Connection (GS):
- Applied immediately after CGRF, this module adaptively filters redundant information inherited from low-quality depth maps.
- It utilizes a channel-wise gating mechanism derived from global average pooling to suppress less relevant channels, ensuring the network focuses on informative cues.
Prediction-Guided Progressive Aggregation (PGPA):
- This decoder module aggregates multi-scale features from coarse to fine resolutions.
- Unlike traditional decoders that simply concatenate features, PGPA uses the saliency prediction from the previous decoding stage as dynamic guidance to modulate current RGB and depth features.
- This progressive strategy helps the network emphasize salient regions and suppress background noise while reducing semantic gaps between feature levels.
Edge Refinement Module (ERM) and Edge Enhancement Module (EEM):
- The ERM processes features from the deepest to the shallowest stages. It incorporates a Cross-Modality Weighting Refinement (CMWR) unit to generate joint attention maps, followed by Point-Fusion and Swin Transformer blocks to model global context.
- The ERM produces four side outputs for deep supervision.
- The EEM acts as a residual convolutional block to strengthen feature representation, specifically preserving boundary details and improving prediction clarity.
- The final prediction is generated by fusing the deepest ERM output with low-level spatial features from a lightweight VGG-based module.

Training Strategy
The model is trained using a deeply supervised hybrid loss function combining Focal Loss, IoU Loss, and SSIM Loss. A dynamic weighting strategy is employed for the SSIM component, increasing its weight as training progresses to emphasize structural similarity in later stages. Deep supervision is applied at four scales to enhance training stability.

Experimental Results
ECFNet was evaluated on eight benchmark datasets (SSD, NLPR, NJU2K, STERE, LFSD, SIP, DUT-RGBD, RGBD135) and the large-scale COME15K dataset. It was compared against 17 state-of-the-art RGB-D SOD methods.

Quantitative Performance: ECFNet ranks among the top three in 19 out of 32 evaluation metrics across all datasets. Specifically, it achieves the first rank on 11 metrics, the second rank on 5 metrics, and the third rank on 3 metrics. It secured the top rank on all four metrics (S-measure, F-measure, E-measure, MAE) for the DUT-RGBD dataset. On the LFSD and SIP datasets, it achieved leading results in S-measure and E-measure.
Generalization: On the COME15K dataset, ECFNet ranked first on all four metrics for the challenging COME-H subset and achieved top performance in three of four metrics for the COME-E subset.
Efficiency: Despite its performance, ECFNet is computationally efficient. It requires 56.1 GFLOPs and 94.6 million parameters, which is significantly lower than recent Transformer-based competitors (e.g., CATNet, EATNet). It achieves an inference speed of 37 FPS on an NVIDIA A100 GPU, substantially outperforming other Transformer-based models.
Ablation Studies: Systematic ablation confirmed the necessity of each component. Removing bidirectional gating, progressive guidance, or edge refinement modules resulted in consistent performance degradation across various metrics, validating the contribution of each module to noise suppression, multi-scale aggregation, and boundary preservation.

Significance and Claims
The paper claims that ECFNet demonstrates that explicitly modeling and preserving modality-specific information, rather than solely learning shared representations, significantly improves RGB-D salient object detection. The proposed framework effectively addresses the challenges of depth noise, modality differences, and boundary ambiguity.

The authors acknowledge that while ECFNet performs competitively, limitations remain in handling ultra-thin structures (e.g., thin wires, antennae), tiny isolated objects, and densely overlapping instances where feature aliasing occurs. They propose that future work will address these by introducing tiny-object auxiliary losses and instance boundary refinement branches, as well as extending the modality-preserving principle to other multi-modal tasks like RGB-thermal SOD and video saliency detection.

More like this