💻 computer science

FG-TreeSeg: Flow-Guided Tree Crown Segmentation without Instance Annotations

This paper introduces FG-TreeSeg, a training-free framework that adapts flow-based delineation from biomedical imaging to achieve robust, instance-level tree crown segmentation in dense remote sensing imagery without requiring manual annotations.

Original authors: Pengyu Chen, Fangzheng Lyu, Sicheng Wang, Cuizhen Wang

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengyu Chen, Fangzheng Lyu, Sicheng Wang, Cuizhen Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a satellite photo of a forest. To a computer, a dense forest often looks like one giant, green blob. The branches of one tree tangle with the next, making it impossible to tell where one tree ends and another begins. This is a huge problem for scientists who need to count trees or measure how much carbon they store.

The paper introduces a new tool called FG-TreeSeg to solve this. Think of it as a "magic pair of glasses" that can instantly separate touching trees without needing a human to teach it what a tree looks like first.

Here is how it works, broken down into simple steps:

1. The Problem: The "Blob" vs. The "Individual"

Usually, computers need thousands of examples where humans have drawn outlines around every single tree to learn how to do it. This is like trying to teach a child to identify apples by showing them 10,000 apples one by one. It takes forever and is expensive.
Other new "smart" computer models (called Foundation Models) are good at general things but get confused in dense forests. They often see a whole group of trees as just one big object.

2. The Solution: Borrowing from Biology

The authors had a clever idea: Trees and biological cells look a lot alike.

The Analogy: Imagine a drop of water on a leaf. No matter how irregular the edge is, if you put a drop of ink in the center, the water flows inward toward that center point.
The Science: In medical imaging, scientists use software to separate touching cells by predicting which way the "water" (or flow) moves. They realized tree crowns are shaped similarly to cells (they are "star-shaped" or "star-convex").
The Trick: Instead of teaching the computer to draw lines, they taught it to predict arrows. Every pixel on a tree crown gets an arrow pointing toward the center of that specific tree.

3. How FG-TreeSeg Works (The Two-Step Dance)

The system works in two stages, like a security guard checking a list before letting people into a party:

Step 1: The "Green Zone" Filter (Semantic Mask)
First, the system uses a pre-trained tool (SegFormer) to simply ask, "Is this green and leafy?" It draws a rough outline of the entire forest canopy and ignores everything else (like roads, grass, or buildings). This stops the system from getting confused by non-tree objects.
- Metaphor: It's like putting a frame around the forest in a photo so the computer only looks inside the frame.
Step 2: The "Flow" Separator (Cellpose-SAM)
Inside that green frame, the system switches to the "cell" logic. It predicts a field of arrows for every pixel.
- If you are on the left side of Tree A, the arrow points to Tree A's center.
- If you are on the right side of Tree B, the arrow points to Tree B's center.
- The Magic: Where two trees touch, the arrows from the left tree point left, and the arrows from the right tree point right. They push against each other, creating a natural "wall" between them. The computer then follows these arrows like a river flowing to a lake, grouping all the pixels that flow to the same center point into one tree.

4. The Results: No Training Required

The best part? The system didn't need to be trained on trees at all.
The authors took a model designed for medical cells and a model designed for general images, combined them, and applied them to forests. They didn't feed it a single example of a tree crown to learn from.

Performance: When they tested it on real forest data (from the NEON and BAMFORESTS datasets), it did surprisingly well.
- It separated touching trees better than some systems that were trained on trees.
- It was almost as accurate as the top "supervised" models (which required massive amounts of human labeling) but did it instantly without any human help.
- It worked on different types of cameras (satellite and drone) and different forest densities.

5. Why This Matters

The paper concludes that while this tool isn't perfect yet (it sometimes needs a tiny bit of human tweaking in very messy forests), it is a game-changer.

The Analogy: Instead of manually drawing a fence around every single tree in a massive forest (which takes years), this tool builds a "flow field" that automatically sorts the trees for you.
The Goal: It's not meant to replace human experts entirely, but to do the heavy lifting. It can quickly create "draft" outlines of trees, which humans can then just clean up. This speeds up the creation of massive datasets needed to train even smarter future AI for forestry.

In short: FG-TreeSeg treats a forest like a crowd of people. Instead of asking "Who is this person?", it asks "Which way are you walking?" and groups everyone walking toward the same center point into a single person. It does this without ever needing to learn what a person looks like beforehand.

Technical Summary: FG-TreeSeg

Problem Statement

Individual tree crown segmentation is a critical task for forest biomass estimation and ecological monitoring but remains a significant bottleneck in remote sensing. The primary challenges stem from the complex spatial structures of forest canopies, where dense, overlapping branches create ambiguous visual boundaries. Existing approaches face two distinct limitations:

Supervised Deep Learning: Methods relying on large-scale instance annotations suffer from high labor costs, subjectivity in ground truth generation, and limited generalization across different sensor types and canopy densities.
Foundation Models: While models like the Segment Anything Model (SAM) offer strong general-purpose segmentation, they often lack specific domain knowledge for forestry. When applied to dense forests, they frequently fail to separate touching instances, leading to under-segmentation. Furthermore, multi-modal approaches incorporating LiDAR or DSMs increase data acquisition costs and computational complexity.

Methodology

The authors propose FG-TreeSeg (Flow-Guided Tree Segmentation), a training-free framework that transfers flow-based delineation techniques from biomedical imaging to remote sensing. The core innovation lies in reconceptualizing tree crown delineation as a topological flow-convergence problem rather than a traditional boundary detection task. The framework leverages the structural isomorphism between biological cells and tree crowns, both of which exhibit star-convex morphological properties.

The pipeline operates in two distinct stages:

1. Canopy Semantic Constraint

To prevent background over-segmentation (e.g., on grass or rooftops) and restrict the search space, the framework employs a pre-trained SegFormer model (MiT-B5 backbone). Trained on the OAM-TCD dataset, this component performs binary semantic segmentation to generate a canopy mask ( $M_{sem}$ ). This mask acts as a spatial prior, isolating valid canopy regions before instance segmentation begins.

2. Flow-Guided Instance Segmentation

The core instance separation is achieved using Cellpose-SAM, adapted for tree crowns. This stage treats segmentation as a dynamical system:

Feature Encoding: A Vision Transformer (ViT) backbone (based on SAM) encodes the input image into a latent space ( $Z$ ), capturing global contextual information and object interior structures.
Flow Prediction: The network predicts a pixel-wise vector field ( $V$ ) representing the gradient of an instance-specific potential surface. Geometrically, these vectors guide pixels inward toward the centroid (sink) of their respective tree crowns.
Flow Convergence: During inference, pixels are treated as particles in a dynamical system. They are iteratively advected along the predicted stream lines using Euler integration ( $p_{\tau+1} = p_\tau + V(p_\tau)$ ). Pixels belonging to the same crown converge to a common stable fixed point, forming the instance mask. This mechanism naturally separates touching instances without requiring post-processing steps like watershed transforms or non-maximum suppression.

Key Contributions

Training-Free Framework: FG-TreeSeg eliminates the need for task-specific annotations or fine-tuning by transferring geometric priors from biomedical imaging (Cellpose-SAM) to remote sensing.
Topological Separation: By modeling tree crowns as star-convex objects within a flow field, the method effectively separates dense, overlapping canopies where traditional boundary-based methods fail.
Hybrid Architecture: The integration of a semantic prior (SegFormer) with a flow-based instance model (Cellpose-SAM) effectively mitigates background noise while ensuring robust instance separation.
Scalability: The approach offers a scalable solution for generating preliminary instance masks, shifting the annotation burden from manual polygon drawing to boundary refinement.

Results

The framework was evaluated on the NEON (Aerial RGB) and BAMFORESTS (UAV VHR) datasets, as well as visually on the TCD dataset.

Quantitative Performance:
- On the NEON dataset, FG-TreeSeg achieved an mAP@50 of 42.30%, outperforming the point-supervised TreePseCo (41.68%) and approaching the supervised DeepForest (49.89%).
- On the BAMFORESTS dataset, the framework achieved an mAP@50 of 67.31%, performing within 2% of fully supervised models like Mask R-CNN (69.05%) and Mask2Former (68.89%) without any manual annotation.
Qualitative Assessment:
- The model demonstrated robust generalization across varying scales (from saplings to heritage trees) and densities (sparse urban to dense forests).
- Visual comparisons on the TCD dataset revealed that while supervised models (e.g., Mask R-CNN trained on TCD) tend to under-segment dense clusters due to annotation biases (aggregating multiple trees into single polygons), FG-TreeSeg successfully separates distinct topological centroids.
- The framework avoids the background sensitivity often seen in raw foundation models like SAM.

Significance and Claims

The paper positions FG-TreeSeg not as a replacement for supervised learning, but as a rapid, out-of-the-box solution for generating preliminary instance masks. Its primary significance lies in:

Bridging the Annotation Gap: It provides a training-free method to generate labels for dense forest environments, significantly accelerating the creation of massive datasets required to train future forestry foundation models.
Robust Generalization: It demonstrates that geometric priors (star-convexity and flow dynamics) can effectively bridge the gap between annotation-free inference and supervised accuracy across diverse sensor types and canopy densities.
Addressing Granularity Mismatches: The authors note that standard quantitative benchmarks may penalize fine-grained predictions in datasets where ground truth aggregates dense clusters. FG-TreeSeg's ability to separate these clusters visually proves its superior fine-grained separability, even when standard metrics are unreliable.

The authors acknowledge that while the current absolute performance is competitive, it may not yet meet the strict thresholds required for precise ecological accounting. However, the framework offers a scalable pathway to improve annotation efficiency and supports the development of more advanced, domain-specific foundation models.