Progressive Per-Branch Depth Optimization for DEFOM-Stereo and SAM3 Joint Analysis in UAV Forestry Applications

This paper presents a progressive per-branch depth optimization pipeline that integrates DEFOM-Stereo disparity estimation, SAM3 segmentation, and a multi-stage filtering scheme to transform noisy stereo imagery into geometrically coherent 3D point clouds, reducing per-branch depth standard deviation by 82% for autonomous UAV-based tree pruning in forestry applications.

The Gist

Imagine you are a tiny robot drone trying to fly through a dense forest and cut specific tree branches without hurting the tree or crashing. To do this safely, the drone needs to know exactly where every single branch is in 3D space, down to the centimeter.

The problem? Trees are messy. Leaves overlap, branches tangle, and the sky looks very different from the wood. When the drone takes a picture with two eyes (stereo vision) to guess depth, the result is often a "noisy" mess—like a photo full of static and blurry spots.

This paper describes a six-step "cleaning factory" that turns a blurry, messy 3D map of a tree into a crystal-clear, precise blueprint of individual branches. The authors call this a "Progressive Pipeline," meaning they didn't try to fix everything at once. Instead, they built a basic version, saw what broke, fixed it, and then fixed the new problems that appeared.

Here is how their "factory" works, using simple analogies:

The Ingredients

1. The Eyes (DEFOM-Stereo): A super-smart AI camera that looks at two photos and guesses how far away everything is. It's great at the big picture but makes small, noisy mistakes.
2. The Tracer (SAM3): Another AI that draws outlines around objects. It tries to draw a line around a specific branch, but sometimes its line is a little too loose, letting in background sky or neighboring leaves.

The Six-Step Cleaning Process

Version 1: The "Naive" Attempt

• The Analogy: You ask a child to trace a branch on a map and then color it in. The child draws the line, but it's a bit wobbly, and they accidentally color in some of the sky behind the branch.
• The Result: The 3D map is full of "ghost" points from the sky and the tree looks like a fuzzy cloud.

Version 2: The "Shrink Wrap" (Morphological Erosion)

• The Fix: To stop the sky from getting in, the team shrinks the traced outline inward, like pulling a rubber band tight around the branch.
• The Problem: This works great for thick tree trunks, but for thin twigs, the rubber band pulls the whole thing apart. The thin branches disappear!

Version 3: The "Skeleton Saver" (Skeleton-Preserving Erosion)

• The Fix: Instead of just shrinking the outline, they find the "spine" or skeleton of the branch first (like finding the backbone of a fish). They keep that spine safe and only trim the fat around it.
• The Result: The thick branches get trimmed of the sky, but the thin twigs stay connected and intact.

Version 4: The "Color Detective" (Color Validation)

• The Problem: Even with a perfect outline, the AI might have grabbed a green leaf that is stuck to the branch. The outline is right, but the color is wrong.
• The Fix: They set up a "color police." They take a sample of the branch's true color (from the very center of the branch) and check every pixel inside the outline. If a pixel is too different in color (like a leaf or a shadow), they kick it out. They also make sure two branches don't claim the same pixel.
• The Result: Now, the mask contains only the branch, no extra leaves or sky.

Version 5: The "Statistical Sweeper" (IQR & Z-Score)

• The Problem: The outline is perfect, but the depth numbers inside are still jittery. Some points say the branch is 1 meter away, and a neighbor says 1.5 meters. It's noisy.
• The Fix: They use standard math rules (like finding the average and throwing out the weird outliers) to smooth things out.
• The Problem: This is a bit blunt. It smooths out the noise, but it also blurs the sharp edges of the branch, making it look like a soft, fuzzy tube.

Version 6: The "Master Sculptor" (The Final Version)

• The Fix: This is the big upgrade. They replace the blunt math with a smarter, five-stage process:
  1. MAD (Median Absolute Deviation): A tougher math rule that ignores extreme outliers better than the old method.
  2. Neighbor Voting: If one point looks weird compared to its neighbors, it gets voted out.
  3. Guided Filtering: This is the magic trick. They look at the color photo again. If the color photo shows a sharp edge (where the branch ends and the sky begins), the depth filter respects that edge and doesn't blur it. It's like using a ruler that only smooths flat surfaces but leaves sharp corners alone.
  4. Adaptive Smoothing: They adjust the "smoothness" based on the branch. Thick trunks get smoothed more; thin twigs get smoothed less so they don't vanish.
• The Result: The final 3D point cloud is incredibly sharp. The branch looks solid, the edges are crisp, and the noise is gone.

Why Does This Matter?

The authors tested this on Radiata Pine trees in New Zealand.

• Before: The 3D data was so noisy (standard deviation of 440mm) that a robot couldn't tell where to cut.
• After: The data became incredibly precise (standard deviation of 31.5mm), a 82% improvement.

The Bottom Line:
This paper isn't just about making pretty pictures. It's about giving a robot the "eyes" and "brain" it needs to safely prune trees without a human holding a chainsaw. By breaking the problem down into six small, logical steps—fixing the outline, checking the color, and then polishing the depth—they turned a messy forest into a clean, digital blueprint ready for autonomous robots.

They even released all their code and data for free, so other scientists can use this "cleaning factory" to help robots work in forests everywhere.

Technical Summary

1. Problem Statement

The paper addresses a critical bottleneck in autonomous UAV-based tree pruning: the inability to generate accurate, centimeter-level 3D reconstructions of individual tree branches in complex forest canopies.

• Context: Radiata pine forestry in New Zealand relies on manual pruning, which poses safety risks. Autonomous UAVs offer a solution but require precise depth data (1–2m operating distance) to position cutting tools.
• The Challenge: While modern foundation models like DEFOM-Stereo (for depth) and SAM3 (for segmentation) are powerful individually, their direct combination fails in forestry due to three specific error families:
  1. Mask Boundary Contamination: SAM3 masks often extend slightly beyond true branch contours, incorporating background (sky) pixels with drastically different depth values.
  2. Segmentation Inaccuracy: Masks may contain color-inconsistent pixels (e.g., attached foliage) or overlap between neighboring branches, leading to ambiguous depth assignments.
  3. Depth Noise: Even high-quality disparity maps contain per-pixel noise, spatial outliers, and boundary artifacts that corrupt per-branch depth histograms and 3D geometry.

2. Methodology: A Progressive Six-Version Pipeline

The authors propose a progressive refinement pipeline that evolves through six versions (V1–V6), where each version diagnoses and corrects specific failure modes of its predecessor.

Foundation Modules

• Data: Stereo pairs (1920×1080) captured by a ZED Mini camera (63mm baseline) on a UAV flying 1–2m from Radiata pine branches.
• Depth Estimation: DEFOM-Stereo (a foundation model using DINOv2 ViT-L) generates dense disparity maps, converted to depth via triangulation ($Z = f \cdot B / D$).
• Segmentation: SAM3 generates instance masks for branches, filtered by confidence ($>0.7$).

Refinement Stages

• Version 1 (Baseline): Direct chaining of DEFOM $\to$ Depth Conversion $\to$ SAM3 $\to$ Point Cloud. Result: High noise and boundary contamination.
• Versions 2 & 3 (Mask Boundary Correction):
  • V2 (Morphological Erosion): Shrinks masks to remove sky pixels. Limitation: Disconnects or destroys thin branches.
  • V3 (Skeleton-Preserving Erosion): A novel topology-aware approach. It erodes the mask but preserves the topological skeleton (centerline) via distance transforms and dilation. This removes boundary noise while guaranteeing thin-branch connectivity.
• Version 4 (Segmentation Accuracy):
  • Color Validation: Uses a LAB-space Mahalanobis distance model built from the mask's core region to reject color-inconsistent pixels (e.g., leaves attached to branches).
  • Overlap Arbitration: Resolves overlapping masks by assigning contested pixels to the branch with the smallest Mahalanobis distance.
  • Result: Clean, non-overlapping masks free of color artifacts.
• Version 5 (Statistical Depth Optimization):
  • Applies a four-phase pipeline: Multi-round IQR outlier removal, Z-score filtering, local spatial detection, and median smoothing.
  • Limitation: IQR has a low breakdown point (25%), and median filtering blurs edges.
• Version 6 (Advanced Robust Depth Optimization - Final):
  • Replaces V5 with a five-stage robust scheme:
    1. MAD Global Detection: Uses Median Absolute Deviation (MAD) (50% breakdown point) for robust global outlier removal, superior to IQR for heavy-tailed noise.
    2. Spatial Density Consensus: Neighbor voting to detect isolated spatial anomalies.
    3. Local MAD Filtering: Local-scale outlier detection insensitive to local outliers.
    4. RGB-Guided Filtering: Uses the RGB image as a structural guide to smooth depth while preserving edges aligned with color transitions.
    5. Adaptive Bilateral Filtering: Tunes the depth-range bandwidth ($\sigma_d$) based on each branch's specific depth variability (MAD), preventing over-smoothing of thin branches.

3. Key Contributions

1. First End-to-End Pipeline: Unifies a stereo foundation model (DEFOM) with instance segmentation (SAM3) specifically for per-branch 3D reconstruction in forestry.
2. Skeleton-Preserving Erosion: An algorithm that removes boundary contamination without sacrificing the topological integrity of thin branches.
3. LAB Mahalanobis Validation: A color-based verification scheme that purges inconsistent pixels and resolves cross-branch overlaps.
4. Five-Stage Robust Depth Pipeline: A novel denoising strategy integrating MAD statistics, spatial consensus, guided filtering, and adaptive bilateral filtering.
5. Systematic Ablation: A rigorous study quantifying the contribution of each refinement stage to final 3D quality.

4. Experimental Results

Evaluated on 1920×1080 imagery of Radiata pine in Canterbury, New Zealand:

• Noise Reduction: The final pipeline (V6) reduced the average per-branch depth standard deviation ($\sigma_Z$) by 82% (from 174.6 mm in V4 to 31.5 mm).
• Outlier Elimination: The depth range (max-min) was reduced from 1545 mm (V1) to 128 mm (V6).
• Geometry Preservation: Unlike V2 (which lost 85.6% of pixels due to naive erosion), V6 preserved 100% of thin-branch structures while removing noise.
• Visual Quality: The resulting 3D point clouds are geometrically coherent, with sharp boundaries and minimal scatter, suitable for autonomous tool positioning.

5. Significance and Impact

• Autonomous Forestry: This work bridges the gap between raw drone imagery and actionable 3D data, enabling the next generation of autonomous tree pruning systems.
• Robustness to Noise: By shifting from IQR to MAD-based statistics and incorporating RGB-guided filtering, the method handles the heavy-tailed noise and occlusion typical of vegetation better than traditional stereo refinement techniques.
• Open Science: The authors have released all code, processed outputs, and interactive 3D visualizations to facilitate further research in UAV forestry.
• Scalability: The pipeline is designed to be adaptable to different tree species and camera configurations, though further validation is needed across diverse environments.

In conclusion, the paper demonstrates that a diagnostic, progressive approach to error correction is superior to monolithic solutions, successfully transforming noisy foundation model outputs into precise, per-branch 3D data required for industrial automation.