WHU-STree: A Multi-modal Benchmark Dataset for Street Tree Inventory

This paper introduces WHU-STree, a comprehensive, multi-modal benchmark dataset featuring synchronized point clouds and high-resolution images of over 21,000 street trees across two cities, designed to overcome limitations in existing datasets by enabling diverse inventory tasks and advancing research in multi-modal fusion and cross-domain generalization for urban tree management.

The Gist

Imagine a city as a giant, living organism. The street trees are its lungs, its umbrellas, and its neighbors. But just like a doctor needs a detailed patient chart to treat a human, city planners need a massive, accurate "health record" for every single tree on the street to manage them properly.

For a long time, creating this record was like trying to count every grain of sand on a beach by hand. Workers had to drive around, stop, climb ladders, measure trunks, and write down species names. It was slow, expensive, and impossible to keep up to date.

Enter WHU-STree, a new "super-tool" for cities, created by researchers at Wuhan University and their partners. Think of it as a giant, digital twin of street trees that helps computers learn how to do the counting and measuring automatically.

Here is a simple breakdown of what makes this paper special, using some everyday analogies:

1. The Problem: The "Blindfolded" Robot

Previously, scientists tried to teach computers to identify trees using data from Mobile Mapping Systems (cars with lasers and cameras on top). But the data they had was like giving a robot a blindfold:

• The "Laser-Only" Blindfold: Some datasets only had 3D laser scans (point clouds). The robot could see the shape of the tree (like a silhouette), but it couldn't see the color or texture of the leaves. It was like trying to identify a person in a dark room just by their shadow.
• The "Camera-Only" Blindfold: Other datasets only had photos. The robot could see the leaves, but it couldn't measure the height or the thickness of the trunk accurately. It was like looking at a photo of a tree and guessing how tall it is without a ruler.
• The "Small Town" Problem: Most old datasets were from just one neighborhood. If you trained a robot on trees in a sunny, warm city, it would get confused when sent to a cold, snowy city.

2. The Solution: The "Super-Visor" (WHU-STree)

The researchers built WHU-STree, which is like giving the robot super-vision.

• Two Eyes, One Brain: They combined 3D Laser Scans (which see the structure) with High-Res 360° Photos (which see the color and texture). Now, the computer can see the tree's skeleton and its skin at the same time.
• The "Cross-City" Training: They collected data from two very different Chinese cities: Nanjing (warm, lush, tall trees) and Shenyang (cooler, different species, shorter trees). This is like training a student not just on one textbook, but on two completely different encyclopedias. This ensures the AI doesn't just memorize one type of tree but learns the concept of a tree, making it ready for any city in the world.
• The Massive Library: They didn't just grab a few trees. They cataloged 21,007 individual trees representing 50 different species. It's like a library that doesn't just have one copy of a book, but thousands of editions, all perfectly organized.

3. What Can This New Tool Do?

With this rich data, the researchers tested how well current AI models could do two main jobs:

• Job A: The "Species Detective" (Classification): Can the AI look at a tree and say, "That's a Maple, not an Oak"?
  • Result: When the AI used both the laser and the photo (multi-modal), it got much better at solving the mystery. It's like a detective who can see the suspect's face and their fingerprints, rather than just one or the other.
• Job B: The "Tree Counter" (Segmentation): Can the AI look at a crowded street and separate one tree from its neighbor, even if their branches are tangled together?
  • Result: This is the hardest part. The AI struggled a bit with tangled branches (like untangling headphones), but the combination of 3D and 2D data helped it make fewer mistakes.

4. The Future: The "City Manager"

The paper doesn't just stop at counting trees. The authors imagine a future where this data powers a Smart City Manager:

• The "Doctor" AI: Imagine an AI that doesn't just count trees, but looks at them and says, "This tree on Main Street is getting sick," or "This tree is too close to the power lines and might fall."
• The "Policy" AI: It could even talk to city planners. A mayor could ask, "Show me all the trees that are too tall for the new bus route," and the AI would instantly generate a list and a map.

The Big Takeaway

WHU-STree is a massive leap forward. It's not just a dataset; it's a training ground for the next generation of AI. By giving computers a "rich" education (combining 3D shapes, 2D photos, and data from different cities), we are teaching them to manage our urban forests better, cheaper, and faster than ever before.

It's the difference between trying to manage a forest with a pencil and paper, versus having a flying drone that can see, measure, and diagnose every single tree in seconds.

Technical Summary

1. Problem Statement

Urban street trees are critical for ecological and social benefits, yet maintaining a comprehensive, accurate, and dynamically updated inventory is challenging. Traditional field surveys are labor-intensive and time-consuming. While Mobile Mapping Systems (MMS) offer an automated alternative using laser scanners and cameras, existing datasets for street tree analysis suffer from three major limitations:

• Small-scale scenes: Most datasets are limited to single regions, hindering the evaluation of algorithm generalization across different urban environments.
• Limited annotation: Existing data often lacks instance-level labels, species information, or morphological parameters, restricting their use for comprehensive asset management.
• Single modality: Current datasets are typically either point-cloud-only (lacking texture/spectral info) or image-only (lacking precise 3D geometry), preventing effective multi-modal fusion for tasks like fine-grained species classification and accurate 3D localization.

2. Methodology: The WHU-STree Dataset

The authors introduce WHU-STree, a large-scale, cross-city, multi-modal benchmark dataset designed to overcome these limitations.

• Data Acquisition:
  • Locations: Data was collected in two distinct Chinese cities: Nanjing (WHU-STree-NJ) and Shenyang (WHU-STree-SY), representing different climatic zones and urban planning styles.
  • Equipment: Two MMS units (Hiscan-Z and Alpha3D) equipped with laser scanners, GNSS/IMU, and 360° panoramic cameras were used.
  • Scale: The dataset covers 66.2 km of road, containing 21,007 annotated tree instances, 50 tree species, and 2 morphological parameters (Height and Diameter at Breast Height - DBH).
• Data Processing & Annotation:
  • Preprocessing: Raw data underwent voxel-based downsampling, adaptive partitioning at road intersections, and ground point removal.
  • Annotation: Trained personnel annotated tree instances and species. 3D annotations were projected onto 2D panoramic images using extrinsic parameters, eliminating the need for separate manual image labeling.
  • Parameter Calculation: Tree height and DBH were derived from point clouds. DBH was estimated by fitting ellipses to trunk point clouds at 1.2–1.4m height, with field measurements used for instances with severe point loss.
• Data Splitting Strategies:
  • Class-balanced split: Uniformly sampled road sections to maintain consistent class ratios for standard training/testing.
  • Cross-city split: Training on one city (e.g., Nanjing) and testing on the other (e.g., Shenyang) to rigorously evaluate cross-domain generalization.

3. Key Contributions

1. Dataset Creation: The first large-scale, multi-modal (Point Cloud + Panoramic Image), cross-city dataset specifically for street tree inventory, featuring 21k+ instances and 50 species.
2. Comprehensive Benchmarking: The authors established baselines for two core tasks:
   • Tree Species Classification: Evaluated voxel-based (MinkNet), point-based (PointMLP, PTv2), and multi-modal (TSCMDL) methods.
   • Individual Tree Segmentation: Evaluated clustering-based (SoftGroup, SegmentAnyTree), transformer-based (SPFormer), and multi-modal panoptic (LCPS) methods.
3. Analysis of Challenges & Opportunities: The paper identifies key research directions, including multi-modal fusion, multi-task collaboration, cross-domain generalization, spatial pattern learning, and the application of Multi-modal Large Language Models (MLLMs) for asset management.

4. Experimental Results

The paper presents extensive benchmark experiments on the WHU-STree-NJ and WHU-STree-SY datasets.

• Tree Species Classification:

  • Best Performer: PTv2 (Transformer-based) achieved the highest Overall Accuracy (87.99%) and mIoU (64.09%) among single-modality methods.
  • Multi-modal Impact: The multi-modal baseline TSCMDL (fusing PointMLP and ResNet) outperformed single-modality PointMLP by 2.49% in mIoU, proving that image texture complements geometric data for species identification.
  • Challenge: Distinguishing visually similar species (e.g., Ligustrum lucidum and Cinnamomum camphora) remains difficult even with multi-modal data.

• Individual Tree Segmentation:

  • Best Performer: SPFormer (Transformer-based) achieved the highest Detection Rate (93.2%) on the Nanjing dataset, benefiting from its end-to-end attention mechanism.
  • Multi-modal Impact: LCPS (LiDAR-camera fusion) showed significant improvements over its 3D-only baseline, particularly in reducing the Commission Rate (false positives) by 7.5% and improving F1-scores by ~4.5% in species-agnostic tasks.
  • Cross-Domain Generalization: Models trained on Nanjing data performed robustly on Shenyang data, with some multi-modal methods (LCPS) even outperforming within-domain results on the target city, suggesting strong generalization capabilities.

• Multi-Task Collaboration:

  • Experiments showed that integrating species classification into the segmentation pipeline (multi-task learning) improved the overall F1-score (from 69.2% to 82.2% on Nanjing), indicating that learning species-specific features aids in better instance delineation.

5. Significance and Future Outlook

• Benchmarking Standard: WHU-STree sets a new standard for evaluating the robustness and generalization of urban tree inventory algorithms, moving beyond single-scene, single-modality limitations.
• Multi-Modal Synergy: The results conclusively demonstrate that fusing 3D geometry with 2D spectral/texture data is essential for high-precision urban tree management.
• Future Directions:
  • Spatial Pattern Learning: Leveraging the regular planting patterns of street trees as prior knowledge to improve segmentation.
  • MLLM Integration: Utilizing Multi-modal Large Language Models to create a "perception-analysis-decision" pipeline, enabling automated risk assessment, maintenance scheduling, and policy-compliant management via natural language queries.
  • Open-Vocabulary Recognition: Addressing the challenge of classifying tree species not present in the training set across different cities.

In conclusion, WHU-STree is a pivotal resource that bridges the gap between deep learning research and practical urban forestry management, facilitating the development of scalable, automated, and intelligent street tree inventory systems.