💻 computer science

SelectAnyTree: A Promptable Instance Segmentation Model for 3D Forest LiDAR Point Clouds

SelectAnyTree is a promptable 3D instance segmentation model for forest LiDAR point clouds that leverages a Click-to-query encoder and CHM-guided initialization to efficiently and accurately delineate individual trees with minimal user interaction, outperforming existing baselines in accuracy, efficiency, and generalization across diverse forest environments.

Original authors: Trung Thanh Nguyen, Daniel Lusk, Kilian Gerberding, Janusch Vajna-Jehle, Tuan-Anh Vu, Duc Viet Le, Tu Vo, Phi Le Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide, Julian Frey, Teja Kattenborn

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Trung Thanh Nguyen, Daniel Lusk, Kilian Gerberding, Janusch Vajna-Jehle, Tuan-Anh Vu, Duc Viet Le, Tu Vo, Phi Le Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide, Julian Frey, Teja Kattenborn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a massive, dense forest. If you look up, you see a tangled mess of branches and leaves. Now, imagine that every single leaf and twig in this forest has been scanned by a laser, turning the whole scene into a giant 3D cloud of dots (points).

The problem scientists face is: How do you tell one specific tree apart from its neighbors in this giant cloud of dots?

Traditionally, computers tried to separate every tree in the forest all at once, like a machine trying to sort a pile of mixed-up Legos in one go. If the machine made a mistake on one tree, you had to throw away the whole pile and start over, or spend hours manually fixing it.

Enter "SelectAnyTree."

Think of SelectAnyTree as a magical "laser pointer" for 3D forests. Instead of forcing the computer to sort the whole forest at once, you simply point and click on the tree you want. The computer then instantly isolates that specific tree, leaving the rest behind.

Here is how it works, using some everyday analogies:

1. The "One-Time Scan" (The Library Card)

Imagine walking into a massive library. Usually, if you want to find a book, you have to ask the librarian to search the whole catalog every time.
SelectAnyTree is different. It scans the entire forest (the library) once and creates a permanent map (a "library card"). Once that map is made, you can ask for any tree, and the computer doesn't need to re-scan the forest. It just looks at its map. This makes it incredibly fast.

2. The "Click-to-Query" (The Magic Wand)

When you click on a tree, the computer doesn't just guess. It uses a special "translator" (the Prompt Encoder).

The Click: You click on a branch.
The Translation: The computer translates your click into a question: "Find the tree that includes this point, but NOT the points I click on as 'no'."
The Result: It pulls out a single, clean 3D shape of just that tree.

3. The "Free Hint" (The Tree Top Hat)

This is the paper's secret sauce. In a dense forest, clicking on a branch can be confusing because branches from different trees overlap.
SelectAnyTree has a built-in "superpower." It looks at the forest from above (like a bird's eye view) to find the very top of the tree (the treetop).

The Analogy: Imagine trying to find a specific person in a crowded room. If you just point to their shoulder, you might grab the wrong person. But if you also point to the top of their head, it's much easier to identify them.
The Benefit: The computer automatically finds the "top of the head" (the treetop) for you without you having to click it. It gives the computer a "free hint" to make sure it grabs the right tree immediately.

4. The "Correction Loop" (The Sculptor)

Sometimes, the computer might accidentally grab a branch from a neighbor.

The Analogy: Think of it like sculpting clay. You start with a rough shape. If you grab too much clay, you just pinch it off.
How it works: If the computer makes a mistake, you simply click on the "wrong" part (a negative click). The computer instantly carves that part away. It learns from your correction in real-time, refining the shape until it's perfect.

Why is this a big deal?

Speed: It is much faster than previous methods. It can isolate a tree with a single click, whereas other models needed five or six clicks to get the same result.
Accuracy: It gets the job done with 78.2% accuracy on the very first click, which is a huge leap forward.
Efficiency: It is lightweight. It doesn't need a supercomputer to run; it's small enough to run on standard equipment, unlike other models that are "heavy" and slow.

In summary:
Before this, finding a specific tree in a 3D forest scan was like trying to find a needle in a haystack by sorting the whole haystack every time. SelectAnyTree lets you just point to the needle, gives you a free hint about where the top of the needle is, and lets you quickly trim away any mistakes. It turns a complex, tedious task into a simple, interactive game of "click and refine."

Technical Summary: SelectAnyTree

Problem Statement

Automated instance segmentation of 3D forest LiDAR point clouds is essential for scalable forest monitoring, biomass estimation, and ecological analysis. However, progress is hindered by the scarcity of labeled data for individual tree instances. Manual annotation is laborious due to the millions of points and complex, overlapping crowns in a single hectare. Furthermore, existing state-of-the-art methods (e.g., ForestFormer3D, ForestMamba) operate in a fully automatic, single-pass mode that segments all trees simultaneously. This design lacks mechanisms for users to select specific trees of interest or interactively correct errors in dense, multi-layered canopies where automatic models frequently fail. While promptable segmentation models exist for 2D images and generic 3D objects (e.g., SAM, Point-SAM), they are not optimized for the specific structural characteristics of forest scenes, such as large-scale sparsity, vertical ground-to-canopy organization, and highly entangled crowns.

Methodology: SelectAnyTree

The authors propose SelectAnyTree, a promptable instance segmentation model designed specifically for 3D forest LiDAR. The model allows users to select and delineate any individual tree using a few clicks, while also supporting interactive error correction. The pipeline consists of four main stages:

3D Scene Encoder: The input point cloud is voxelized and processed once by a sparse encoder (based on state-space models) to generate reusable per-voxel features. An auxiliary branch predicts tree/non-tree logits and instance-discriminative embeddings to maintain backbone health during training.
Prompt Encoder: This component converts a set of user clicks (positive or negative) into a single content query. It fuses three elements:
- Geometric Anchor: A pooled local backbone feature from a cylinder around positive clicks, capturing local crown context.
- Positional and Label Encoding: Normalized click coordinates and polarity (positive/negative) are embedded using random-Fourier encoding and a learned embedding.
- Signed Aggregation: Click tokens are combined via a signed mean, where positive clicks pull the query toward the target and negative clicks push it away.
  The final query is a linear projection of the aggregated tokens added to the geometric anchor.
Query Decoder: A state-space query decoder (utilizing linear-time complexity) takes the prompt query and cached voxel features. It iteratively refines the query through local $k$ -nearest-neighbor aggregation and state-space modeling to predict a voxel-resolution mask and an objectness score. The mask is then mapped back to the original point cloud.
CHM-Guided First Prompt & Interactive Rollout:
- CHM Prompt: To bridge automatic and interactive modes, the model optionally uses a Canopy Height Model (CHM) treetop as a "free" first prompt. This geometry-guided prompt consists of a positive click at the treetop peak and another at the trunk interior, requiring no user input.
- Shared Rollout: Both training and evaluation utilize an identical "interactive click-simulation" rollout. Starting from a ground-truth mask, the system simulates user corrections by sampling clicks from false-negative (missed) or false-positive (leaked) regions. This ensures the optimization objective matches the evaluation protocol exactly.

Key Contributions

SelectAnyTree Model: The first promptable instance segmentation model tailored for 3D forest LiDAR, enabling users to select and refine individual trees with few clicks and correcting errors in dense canopies without retraining.
Forest-Aware Architecture:
- A lightweight prompt encoder that fuses click position, polarity, and local backbone features into a single query, effectively disambiguating trees in entangled canopies.
- Integration of the CHM treetop as a geometry-guided, zero-cost first prompt that anchors the segmentation process.
Consistent Training and Evaluation: Adoption of a shared interactive click-simulation rollout for both training and testing, ensuring that the model is optimized for the specific interactive refinement task.
Efficiency: The model encodes the scene once and reuses features for all prompts, allowing for efficient multi-tree selection with linear-time complexity.

Experimental Results

The model was evaluated on the FOR-instanceV2 benchmark (seven diverse forest regions) and an independent held-out dataset (LAUTx).

Interactive Performance: SelectAnyTree achieved 78.2 IoU with a single user click, outperforming the strongest promptable baseline (Point-SAM) by 24.8 points. It reached all accuracy targets (IoU 0.7, 0.8, 0.9) with the fewest clicks compared to baselines.
Generalization: The model demonstrated strong generalization to the unseen LAUTx dataset, maintaining high single-click accuracy and low click counts, whereas other promptable baselines degraded significantly.
Efficiency: SelectAnyTree is the most compact model (19.4M parameters) and the fastest. It processes 640k points per patch (vs. 20k–40k for baselines) and completes a 5-click session in 549 ms, which is over 4x faster than the next fastest baseline and 16x faster than Point-SAM. It also uses significantly less GPU memory (1.9 GB vs. 22.4 GB for Point-SAM).
Zero-Click Setting: When prompts were automatically grid-sampled (zero-click setting), SelectAnyTree achieved the best Precision and F1 among promptable methods, though it trailed specialized fully automatic models (e.g., ForestMamba), as expected.

Significance and Claims

The paper claims that SelectAnyTree addresses the fundamental limitation of current forest intelligence: the lack of human control and the high cost of annotation. By bringing the "segment anything" paradigm to 3D forests, it enables:

Interactive Annotation: Users can select specific trees or correct errors in dense canopies without processing the entire scene or retraining the model.
Label Efficiency: The ability to generate high-quality masks from a few clicks reduces the labor burden of creating ground-truth data for downstream tasks like species classification and biomass estimation.
Scalability: The model's efficiency and ability to generalize across diverse forest types suggest it can support large-scale, crowd-sourced label generation efforts, potentially lifting the label bottleneck that currently constrains LiDAR-based forest monitoring.

The authors acknowledge limitations, noting that the CHM-dependent treetop prompt may be less reliable in extremely dense, multi-layered canopies where tree-voxel prediction degrades, and that fixed allometric parameters may not transfer perfectly across all forest types. Future work includes exploring richer prompt types (boxes, scribbles) and large-scale pre-training.