💻 computer science

Granularity-Aware Transfer for Tree Instance Segmentation in Synthetic and Real Forests

This paper introduces MGTD, a mixed-granularity dataset and a four-stage protocol featuring granularity-aware distillation to effectively transfer structural priors from fine-grained synthetic tree annotations to coarse-label real-world data, significantly improving instance segmentation performance for small and distant trees.

Original authors: Pankaj Deoli, Atef Tej, Anmol Ashri, Anandatirtha JS, Karsten Berns

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pankaj Deoli, Atef Tej, Anmol Ashri, Anandatirtha JS, Karsten Berns

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a forest. To do this, the robot needs to "see" individual trees, distinguishing their trunks from their leafy crowns, so it doesn't crash into them.

The problem is that real forests are messy. They have changing light, shadows, and leaves everywhere. To teach a robot, you'd normally need to take thousands of photos of real trees and manually draw a perfect outline around every single trunk and leaf. This is incredibly expensive, slow, and boring for humans.

So, scientists usually use video games (synthetic data) to teach robots. You can generate thousands of perfect forest images in a computer, and the computer already knows exactly where every tree part is. But here's the catch: Robots trained on video games often fail in real life. The trees look too perfect, the lighting is too even, and the robot gets confused when it sees a real, messy forest.

This paper introduces a clever solution to two big problems:

The "Game vs. Reality" Gap: The robot doesn't recognize real trees because they look different from the game trees.
The "Detail vs. Simplicity" Gap: In the game, the robot is taught to see two things: the trunk and the whole tree. In the real world, humans only label it as one thing: "Tree." The robot gets confused because the rules changed.

The Solution: The "Master Chef" and the "Apprentice"

The authors created a new dataset called MGTD (Mixed-Granularity Tree Dataset). Think of this as a massive library containing:

53,000 perfect "Game" images where every tree trunk and crown is labeled separately (Fine-grained).
3,600 real-world images where trees are just labeled as "Tree" (Coarse-grained).

To bridge the gap, they invented a method called Granularity-Aware Distillation. Here is the analogy:

1. The Two Master Chefs (The Teachers)

First, they train two expert "Master Chefs" using the perfect Game data:

Chef Trunk: An expert at spotting just the vertical tree trunks.
Chef Crown: An expert at spotting the whole tree shape (trunk + leaves).
These chefs are amazing at their jobs, but they only know how to cook in the "Game Kitchen." If you put them in a "Real Forest Kitchen," they get overwhelmed and make mistakes.

2. The Apprentice (The Student)

Next, they train an "Apprentice" using only the Real Forest images. This Apprentice knows what real forests look like (the messy lighting, the dirt), but it's a bit clumsy because it only has simple labels. It doesn't know how to separate a trunk from a leafy branch effectively.

3. The Secret Sauce: Logit-Space Merging

This is the magic part. Instead of just letting the Apprentice watch the Masters, they use a special technique to merge the Masters' knowledge into the Apprentice's brain without forcing the Apprentice to learn the complex "Game" labels.

Imagine the Masters are shouting instructions:

Chef Trunk says: "There is a vertical line here!"
Chef Crown says: "There is a big bushy shape here!"

Instead of making the Apprentice learn two separate languages, the system blends these shouts into a single, clear instruction: "There is a Tree here."

They do this mathematically (using something called "logit-space merging"), which is like taking the best parts of both chefs' opinions and combining them into one confident guess. This allows the Apprentice to learn the structural secrets of the trees (like how trunks are straight and crowns are round) from the Masters, while still learning to recognize the real-world look of the forest.

The Results: A Lightweight Robot that Outperforms Giants

The most surprising result?

The "Apprentice" (a smaller, simpler AI model) trained with this method became better than "Giant" models trained only on real data.
Usually, you need a massive, super-complex computer brain to see small, distant, or hidden trees. But because this Apprentice learned the structural secrets from the Masters, it could spot tiny, thin trunks in the distance that the Giant models missed.

Why This Matters

This isn't just about trees. It's a new way of teaching AI.

Old Way: "Here is a picture of a car. Here is a picture of a truck. Learn the difference." (Requires perfect labels).
New Way: "Here is a perfect simulation of a car engine and a wheel. Here is a messy real photo of a vehicle. Learn to combine the engine knowledge with the real photo."

This approach allows robots to learn from cheap, easy-to-make simulations and apply that knowledge to expensive, hard-to-label real-world tasks, making autonomous robots (like self-driving cars or forest explorers) much smarter and more reliable.

1. Problem Statement

The paper addresses a critical bottleneck in forestry robotics: the Synthetic-to-Real (Sim→Real) domain gap, specifically compounded by a label granularity mismatch.

The Domain Gap: Models trained on synthetic data often fail to generalize to real-world forest scenes due to differences in illumination, texture, and background clutter.
The Granularity Mismatch: Synthetic datasets typically offer fine-grained annotations (separating "tree trunks" from "whole trees"), whereas real-world datasets are often limited to coarse annotations (a single "Tree" class) due to the high cost and complexity of manual fine-grained labeling.
The Challenge: Existing domain adaptation methods assume identical class definitions between source and target domains. This paper tackles the scenario where the source (synthetic) has fine-grained labels and the target (real) has coarse labels, a setting where standard transfer learning fails because the structural priors learned in simulation cannot be directly mapped to the coarse real labels without specific adaptation.

2. Methodology

The authors propose a four-stage evaluation protocol and a novel Granularity-Aware Knowledge Distillation framework.

A. The MGTD Dataset

To study this problem systematically, the authors introduced MGTD (Mixed-Granularity Tree Dataset):

Synthetic Portion: 53,521 images generated in Unreal Engine 5 under four illumination conditions (dawn, dusk, noon, snow). Annotated with fine-grained masks for Tree Trunks and Whole Trees.
Real Portion: 3,600 ground-level images captured in mixed forests. Annotated only with a single "Tree" label (coarse granularity).
Split: Both portions follow a 70/20/10 train/validation/test split.

B. Four-Stage Evaluation Protocol

The methodology isolates specific challenges through four phases:

Phase 1 (Synthetic Upper Bound): Train two separate "Teacher" models on synthetic data: $T_{trunk}$ (for trunks) and $T_{whole}$ (for whole trees). This establishes the theoretical performance ceiling in the synthetic domain.
Phase 2 (Zero-Shot Transfer): Evaluate the synthetic teachers directly on real data using a label-agnostic metric (union of trunk and whole-tree predictions matched against real "Tree" ground truth). This quantifies the pure domain gap.
Phase 3 (Real-Only Baseline): Train a "Student" model solely on real data with the coarse "Tree" label. This establishes the baseline performance achievable without synthetic data.
Phase 4 (Granularity-Aware Distillation): The core contribution. Transfer knowledge from the frozen synthetic teachers to the real-data student.

C. Granularity-Aware Distillation Mechanism

The distillation process bridges the fine-to-coarse label gap without re-labeling real data:

Dual Teachers: Utilizes both $T_{trunk}$ and $T_{whole}$ to capture complementary structural cues (vertical stem geometry vs. global canopy silhouette).
Logit-Space Merging: Instead of merging masks or probabilities (which assumes independence), the method merges the classification logits ( $z$ ) from both teachers:
$z_{tree} = \log(e^{z_{trunk}} + e^{z_{whole}})$
This creates a unified "Tree" logit that preserves contributions from both granularities without making independence assumptions.
Mask Unification: Teacher masks are merged via set union ( $M_{tree} = M_{trunk} \cup M_{whole}$ ) to provide unified geometric signals.
Loss Function: The student is optimized using a joint objective combining supervised loss on real data and Knowledge Distillation (KD) losses across classification, bounding box, and mask heads:
$L = L_{sup} + \lambda_{KD}(L_{KD}^{cls} + L_{KD}^{box} + L_{KD}^{mask}) + \lambda_{cons}L_{cons}$
Where $L_{cons}$ enforces consistency under augmentation (Mean Teacher framework).

3. Key Contributions

MGTD Dataset: The first large-scale benchmark pairing fine-grained synthetic forest data with coarse-grained real forest data, enabling controlled study of mixed-granularity transfer.
Four-Stage Protocol: A systematic framework to disentangle the effects of domain shift (appearance) from granularity mismatch (label structure).
Granularity-Aware Distillation: A novel method to transfer fine-grained structural priors from synthetic teachers to a coarse-label student via logit-space merging, proving that structural knowledge can compensate for missing real-world annotations.
Efficiency Insight: Demonstrated that a lightweight student model (ResNet-50) enhanced by distillation outperforms much larger models (ResNeXt-101, Swin-T) trained solely on real data.

4. Experimental Results

Experiments were conducted using Mask R-CNN (with ResNet, ResNeXt, and Swin-T backbones) and YOLO variants.

Phase 1 (Synthetic Performance): Teachers achieved high performance on synthetic data (e.g., Swin-T Whole Tree mAP@0.5:0.95 = 0.880).
Phase 2 (Zero-Shot Gap): Direct transfer resulted in catastrophic performance drops (e.g., Whole Tree mAP dropped from 0.880 to 0.066, a >90% relative drop), confirming the severity of the domain gap.
Phase 3 (Real Baseline): Training on real data alone improved performance significantly (Swin-T mAP@0.5:0.95 = 0.420) but struggled with small, distant, or occluded trees due to the lack of structural priors.
Phase 4 (Distillation Success):
- The distilled ResNet-50 student achieved 0.429 mAP (Segm), outperforming the ResNeXt-101 (0.363) and Swin-T (0.420) real-only baselines.
- Qualitative Gains: The distilled model showed significantly improved recall for thin, distant, and partially occluded trunks that were missed by the real-only model.
- Structural Consistency: The model produced masks with better vertical alignment and structural continuity, effectively transferring the "shape priors" from the synthetic domain.

5. Significance and Implications

Redefining Sim→Real: The paper argues that the primary barrier in forestry perception is not just appearance shift, but the loss of structural information due to coarse labeling.
Data Efficiency: It demonstrates that the value of synthetic data lies not just in volume, but in the quality of structural supervision (fine-grained labels) which can be distilled into real-world models.
Broader Applicability: This approach offers a solution for any domain where fine-grained simulation is cheap but real-world fine-grained annotation is expensive (e.g., medical imaging, industrial defect detection, agriculture).
Model Efficiency: It challenges the assumption that larger models are always better for real-world tasks, showing that better supervision strategies (distillation) can yield superior results with smaller, more deployable architectures.

In conclusion, the paper establishes that by explicitly addressing label granularity mismatch through logit-space merging and dual-teacher distillation, it is possible to create robust, high-performance forestry perception systems that leverage the structural richness of synthetic data while adhering to the practical constraints of real-world data collection.