EDFNet: Early Fusion of Edge and Depth for Thin-Obstacle Segmentation in UAV Navigation

This paper introduces EDFNet, a modular early-fusion framework that integrates RGB, depth, and edge information to enhance thin-obstacle segmentation for UAV navigation, demonstrating that pretrained RGB-Depth-Edge fusion with a U-Net backbone achieves the best balance of performance and efficiency while highlighting that ultra-thin structure detection remains an open challenge.

The Gist

Imagine you are flying a drone through a dense forest. Your goal is to get from point A to point B without crashing. The big trees are easy to see; they are huge and obvious. But what about the invisible killers? The thin power lines, the delicate spiderwebs, the slender branches, and the telephone wires?

To a human eye, these are hard to spot. To a standard camera, they are almost invisible. They are so thin they might only cover a single pixel on your screen, and they often blend right into the background. If your drone misses even one of these, it could snap the wire, crash, and the mission is over.

This paper introduces EDFNET, a new "brain" for drones designed specifically to see these invisible dangers. Here is how it works, explained with some everyday analogies.

The Problem: The "Needle in a Haystack"

Standard drone cameras are like a person looking at a haystack trying to find a needle. If you only look at the color (RGB), the needle might look exactly like the hay. If you only look at depth (how far away things are), the needle might be too thin for the sensor to register. If you only look at edges (where things start and stop), the needle might be too faint to trace.

Because these thin objects are so rare compared to big things like trees or buildings, computer models often ignore them, thinking, "Oh, that's just a tiny speck of noise," and delete it.

The Solution: EDFNET (The "Super-Senses" Approach)

The authors created a system called EDFNET. Think of EDFNET not as a single camera, but as a super-sensory team that fuses three different types of information right at the very beginning, before the brain even starts thinking.

They combine three "senses":

1. RGB (The Eyes): What the object looks like (color and texture).
2. Depth (The Sonar): How far away the object is (geometry).
3. Edge (The Outline): A sketch of where the object's boundaries are.

The "Early Fusion" Analogy:
Imagine you are trying to identify a suspect in a crowd.

• Late Fusion (The old way): You ask one person to describe the face, another to describe the height, and a third to describe the outline. Then, at the end, you try to combine their notes. By then, you might have missed the connection.
• Early Fusion (EDFNET's way): You give all three clues to a single detective at the same time. The detective looks at the face, the height, and the outline simultaneously from the very first second. This allows them to spot the "thin" suspect much faster and more accurately.

EDFNET takes the color image, the depth map, and the edge sketch, stacks them together like a sandwich, and feeds this "super-sandwich" into the computer's brain immediately.

The Experiment: The "Driving Test"

The researchers tested this new brain on a dataset called DDOS (Drone Depth and Obstacle Segmentation). It's like a driving school for drones, filled with pictures of wires, poles, and branches.

They tested 16 different combinations:

• Different "brains" (U-Net and DeepLabV3).
• Different "sensory inputs" (Just eyes? Eyes + Sonar? Eyes + Outline? All three?).
• Different training methods (Did the brain study a textbook first? Or did it learn from scratch?).

The Results: What Worked?

The results were a mix of "Great success" and "Still a work in progress."

The Winner:
The best performer was the Pre-trained U-Net with all three senses (RGB + Depth + Edge).

• Analogy: Think of this as a student who studied a textbook (pre-trained) and then took a test with a magnifying glass, a ruler, and a highlighter (all three senses).
• Performance: It was the most accurate at finding the thin wires and poles, and it did so quickly enough to be useful for a real drone (about 19 frames per second, which is like watching a smooth video).

The Good News:
Adding the depth and edge information definitely helped. It made the drone much better at noticing the edges of things and remembering to look for them (Recall). It was less likely to crash into a wire it missed.

The Bad News (The "Ultra-Thin" Problem):
Even with the super-senses, the system still struggled with the rarest, thinnest objects (like a single, very fine wire).

• Analogy: It's like trying to see a single strand of hair in a hurricane. Even with the best tools, if the hair is too thin and the background is too messy, the computer still misses it.
• The paper admits that while EDFNET is a great baseline, perfectly detecting the "ultra-thin" category is still a huge, unsolved challenge.

Why This Matters

This paper doesn't claim to have solved the problem of flying through a forest of invisible wires forever. Instead, it provides a solid, practical foundation.

It proves that if you want a drone to see thin obstacles, you shouldn't just rely on a camera. You need to combine color, distance, and outlines right from the start. EDFNET is a simple, modular, and effective way to do that. It's a "good enough" system that works well today, and it sets the stage for future engineers to build even smarter systems that can eventually see those invisible strands of wire perfectly.

In short: EDFNET is the drone's new pair of glasses that combines sight, depth, and outlines to stop it from crashing into things it can't see. It's not perfect yet, but it's a massive step up from what we had before.

Technical Summary

1. Problem Statement

Autonomous Unmanned Aerial Vehicles (UAVs) face significant safety risks when navigating environments containing ultra-thin obstacles (e.g., power lines, wires, branches, poles). These structures are notoriously difficult to perceive due to:

• Minimal Pixel Footprint: They occupy very few pixels in an image.
• Weak Visual Contrast: They often blend into complex backgrounds.
• Class Imbalance: Obstacle pixels are vastly outnumbered by background pixels, causing standard models to underrepresent them.
• Limitations of Existing Methods: Current semantic segmentation models often rely on RGB data alone or use late-fusion strategies for multimodal data. These approaches struggle to preserve fine, elongated structures because pooling and downsampling suppress weak local evidence. Furthermore, existing datasets often focus on specific infrastructure (like power lines) rather than general thin obstacles in cluttered scenes.

2. Methodology: EDFNET

The authors propose EDFNET (Edge–Depth Fusion Network), a modular early-fusion segmentation framework designed to integrate RGB, depth, and edge information at the input level.

• Core Architecture:

  • Early Fusion Strategy: Instead of processing modalities in separate branches or fusing them later, EDFNET concatenates RGB, Depth, and Edge maps along the channel dimension before the first convolutional layer. This allows the backbone to learn joint appearance, geometric, and boundary representations from the outset.
  • Backbones: The framework is evaluated using two standard segmentation architectures: U-Net (encoder-decoder with skip connections for spatial detail) and DeepLabV3 (context-oriented with Atrous Spatial Pyramid Pooling).
  • Modality Configurations: The system tests four input combinations:
    1. RGB: Standard 3-channel image.
    2. RGBD: RGB + Normalized Depth.
    3. RGBE: RGB + Edge Map.
    4. RGBDE: RGB + Depth + Edge (5-channel input).

• Input Construction & Preprocessing:

  • Edge Extraction: Edge maps are generated from grayscale RGB images using the Sobel operator.
  • Depth Normalization: Depth maps are rescaled to $[0, 1]$ per frame to mitigate sensor scale variations.
  • Contrast Enhancement: RGB images undergo CLAHE (Contrast Limited Adaptive Histogram Equalization) to improve visibility in low-contrast regions.
  • Augmentation: Synchronized augmentation (flipping, cropping, affine transforms) is applied to all modalities to preserve spatial alignment.

• Training Procedure:

  • Loss Function: A class-weighted cross-entropy loss is used to address severe class imbalance, giving higher weight to rare thin-obstacle classes.
  • Optimization: Adam optimizer with an initial learning rate of $5 \times 10^{-4}$, batch size of 16, and 50 epochs.
  • Pretraining: Experiments compare models trained from scratch versus those initialized with ImageNet weights (ResNet-34 for U-Net, ResNet-50 for DeepLabV3).

3. Key Contributions

1. EDFNET Framework: A modular, early-fusion architecture that integrates RGB, depth, and edge cues with minimal architectural changes to standard backbones.
2. Systematic Evaluation: A comprehensive study of 16 configurations (4 modality types $\times$ 2 backbones $\times$ 2 pretraining settings) on the DDOS (Drone Depth and Obstacle Segmentation) dataset.
3. New Metric (TSE): Introduction of the Thin-Structure Evaluation Score (TSE), a composite metric designed to prioritize safety-critical performance:
   $$TSE = 0.45 \times \text{bIoU} + 0.30 \times \text{Recall} - 0.15 \times \text{FPR} + 0.10 \times \text{mIoU}$$
   This formulation emphasizes boundary fidelity and recall while penalizing false positives.
4. Empirical Insights: Demonstration that early fusion provides consistent gains in boundary-sensitive and recall-oriented metrics, particularly for thin structures.

4. Experimental Results

The model was evaluated on the DDOS test split. Key findings include:

• Best Performing Model: The Pretrained RGBDE U-Net achieved the highest overall performance:

  • TSE: 0.244 (Highest)
  • Mean IoU (mIoU): 0.219
  • Boundary IoU (bIoU): 0.234
  • Recall: 0.404
  • False Positive Rate (FPR): 0.026 (Lowest)
  • Runtime: 19.62 FPS (19.62 frames per second).

• Impact of Modalities:

  • Early fusion of RGB, Depth, and Edge (RGBDE) generally outperformed single-modality or dual-modality inputs, especially when paired with a pretrained U-Net.
  • Gains were most pronounced in boundary fidelity (bIoU) and recall, rather than raw overlap (mIoU) alone.
  • Pretraining significantly improved performance for RGB and RGBDE configurations but degraded performance for RGBE (Edge-only augmentation) in U-Net, suggesting edge cues may conflict with ImageNet priors in certain contexts.

• Limitations:

  • Ultra-thin Categories: Despite improvements, performance on the rarest "Ultra-thin" classes remained extremely low (best IoU $\approx$ 0.007).
  • Failure Modes: Models still struggle with fragmented predictions on low-contrast wires, missed detections in clutter, and false positives on high-frequency textures (e.g., foliage).

• Efficiency: All configurations operated within a narrow efficiency band (17.59 – 20.81 FPS), indicating that adding depth and edge channels does not impose a significant runtime penalty on standard hardware.

5. Significance and Future Work

• Significance: The paper establishes early RGB-Depth-Edge fusion as a practical, modular, and computationally efficient baseline for thin-obstacle segmentation. It highlights that while standard metrics (mIoU) may not fully capture the difficulty of thin structures, boundary-sensitive metrics (bIoU) and recall are better indicators of safety-critical performance.
• Open Challenge: The consistent failure to reliably segment the rarest ultra-thin categories indicates that simple early fusion is not a complete solution; more advanced techniques are needed.
• Future Directions:
  • Modeling: Moving beyond simple channel concatenation to adaptive fusion (attention/gating mechanisms) and boundary-aware supervision.
  • Deployment: Profiling on embedded UAV hardware (SWAP constraints) and optimizing for real-world sensing imperfections (motion blur, noise).
  • Evaluation: Testing in closed-loop navigation simulations to verify if segmentation gains translate to actual collision avoidance.

In conclusion, EDFNET provides a robust foundation for UAV perception in cluttered environments, proving that integrating geometric and boundary cues early in the pipeline significantly enhances the detection of safety-critical thin obstacles, even if the ultimate challenge of ultra-thin segmentation remains unsolved.