TreeON: Reconstructing 3D Tree Point Clouds from Orthophotos and Heightmaps

TreeON is a novel neural framework that reconstructs detailed 3D tree point clouds from a single orthophoto and Digital Surface Model by leveraging a synthetic dataset and a unique training strategy combining geometric, shadow, and silhouette losses to achieve high-quality, generalizable results without requiring ground truth data or species labels.

The Gist

Imagine you are looking at a map of a forest from a satellite. You can see the tops of the trees (the "canopy") and you can see how high they are, but you can't see the branches, the leaves, or the 3D shape of the tree. It's like looking at a flat, green puddle on a piece of paper.

TreeON is a new computer program that takes this flat, 2D "puddle" and magically turns it back into a detailed, 3D tree, complete with branches and leaves, just by looking at two simple things:

1. A top-down photo (like a bird's-eye view).
2. A height map (a digital version of the ground showing how high the trees stick up).

Here is how it works, explained with some everyday analogies:

1. The Problem: The "Flat Earth" of Trees

Usually, to make a realistic 3D tree, you need a laser scanner on the ground (LiDAR) or dozens of photos taken from every angle. But in huge rural areas, we don't have that. We only have flat maps.

• The Analogy: Imagine trying to rebuild a complex sculpture of a tree just by looking at its shadow on the ground and a flat photo of its top. Most computers would just draw a simple cone or a blob. They would miss the messy, beautiful details of the branches.

2. The Solution: The "Digital Gardener"

TreeON is like a super-smart digital gardener who has memorized thousands of trees. It doesn't need to know the specific name of the tree (like "Oak" or "Pine"). It just looks at the clues.

• The Clues:
  • The Height Map (DSM): This tells the computer, "This tree is 10 meters tall and 5 meters wide." It's the skeleton.
  • The Photo (Orthophoto): This tells the computer, "The top looks textured, and there's a shadow stretching to the left." It's the skin and the mood.

3. How It Learns: The "Video Game Simulator"

You can't teach a computer to guess tree shapes using real trees because we don't have 3D scans of every tree in the world. So, the creators built a video game simulator.

• The Analogy: They created a virtual forest with 3,000 fake trees. They used a robot to "photograph" these fake trees from above and measure their heights, creating a perfect training set.
• The computer (TreeON) plays a game: It looks at the fake photo and height map, tries to guess what the 3D tree looks like, and then checks its answer against the real 3D model. If it's wrong, it learns. It does this millions of times until it becomes an expert at guessing.

4. The Secret Sauce: "Shadow and Silhouette" Training

This is the most clever part. Usually, computers just try to match the shape. TreeON uses a special training trick called "Shadow and Silhouette Loss."

• The Analogy: Imagine you are trying to guess what a person looks like just by seeing their shadow on a wall.
  • Silhouette: The computer checks if the outline of its guessed tree matches the outline of the real tree from the side.
  • Shadow: The computer checks if the shadow its guessed tree would cast matches the actual shadow seen in the photo.
• If the computer guesses a tree that is too round, the shadow won't match the photo. If it guesses a tree that is too spiky, the outline won't match. By checking these "shadows" and "outlines," the computer is forced to build a tree that looks and acts like a real one, even though it's just guessing.

5. The Result: A Lightweight 3D Forest

Once trained, TreeON can take a flat map of a real forest and instantly generate a 3D forest.

• Speed: It's incredibly fast. It can build a 3D tree in 0.3 seconds.
• Size: The file is tiny (like a text message), whereas high-quality 3D models are usually huge files (like a movie).
• Quality: The trees look natural. They have the right height, the right width, and even the right "messy" branch structure.

Why Does This Matter?

Think of digital maps (like Google Earth) as a 2.5D world. You can see the ground, but if you zoom in on a tree, it looks like a flat green blob.
TreeON allows us to turn those flat blobs into immersive 3D forests without needing expensive lasers or teams of photographers. It makes digital maps feel real, helping us navigate, plan cities, or just enjoy a virtual walk in the woods, all while keeping the computer running fast and light.

In short: TreeON is a magic trick that turns a flat shadow and a height chart into a full, detailed 3D tree, using a "shadow-checking" game it learned in a video game simulator.

Technical Summary

Here is a detailed technical summary of the paper "TreeON: Reconstructing 3D Tree Point Clouds from Orthophotos and Heightmaps".

1. Problem Statement

Digital 3D maps are essential for environmental monitoring and navigation, yet high-resolution 3D models of vegetation in rural and natural environments are scarce. Existing methods face a trade-off:

• Data-Rich Methods: Techniques using terrestrial LiDAR, multi-view photogrammetry, or species-specific procedural modeling produce high-fidelity results but are computationally expensive, require specialized data collection, and do not scale to large areas.
• Data-Sparse Limitations: Rural mapping typically relies on Orthophotos (top-down aerial imagery) and Digital Surface Models (DSMs) (heightmaps). However, these inputs only provide 2.5D representations, lacking the fine structural details (branches, foliage volume) necessary for realistic close-up views.
• The Gap: There is a need for a lightweight, automated, and scalable method to reconstruct structurally plausible and visually appealing 3D tree point clouds using only sparse top-down geodata (a single orthophoto and its corresponding DSM), without requiring species labels, ground scans, or multi-view imagery.

2. Methodology: TreeON Framework

TreeON is a neural-based framework that reconstructs 3D tree point clouds by fusing geometric cues from DSMs and visual cues from orthophotos.

A. Synthetic Training Dataset Generation

Since paired ground-truth 3D tree data with orthophotos/DSMs is unavailable in the real world, the authors created a synthetic dataset:

• Source: Procedurally generated tree meshes using the Grove add-on in Blender (covering 17 species, 60% coniferous, 40% deciduous).
• Sampling: Meshes are converted to colored point clouds (~6,000 points) via Poisson-disk sampling.
• Rendering:
  • DSM: Generated by rendering per-pixel depth from a top-down orthographic camera, normalized to a heightmap.
  • Orthophoto: Rendered with directional lighting to capture canopy texture and cast shadows. Sun positions vary in azimuth and elevation to diversify shadow cues.
  • Environment: Trees are placed on diverse terrain patches (flat, hilly, forested) to ensure context awareness.

B. Network Architecture

The model uses a multi-modal encoder-decoder architecture to predict occupancy and color at arbitrary 3D query points:

1. Inputs:
   • Orthophoto: Encoded via a ResNet-18 backbone (pre-trained on ImageNet) to extract texture and shadow features.
   • DSM: Encoded as a sparse point cloud via a PointNet-style architecture to capture 3D geometric structure.
   • Query Points: 3D coordinates sampled within a unit cube, encoded using Fourier positional encoding.
2. Fusion: The image and point cloud embeddings are concatenated into a 2048-dimensional latent vector.
3. Decoder: A deep Multi-Layer Perceptron (MLP) takes the fused latent vector and query points to predict:
   • Occupancy: Whether a point is inside or outside the tree.
   • Color: RGB values for the point.
4. Refinement Module: A lightweight residual MLP refines the coarse occupancy logits to recover sharp boundaries (e.g., fine branches) that the main decoder might smooth over.

C. Supervision Strategy

The training process utilizes a novel combination of five loss terms to guide the network without species labels:

1. Binary Cross-Entropy (BCE) Loss: Enforces point-level occupancy accuracy (inside vs. outside).
2. Color Loss (L2): Matches predicted RGB values to the ground truth.
3. Shadow Loss: Uses a differentiable renderer to project the predicted 3D shape into a top-down shadow map. This is compared against the ground-truth shadow using LPIPS (Learned Perceptual Image Patch Similarity) to ensure the 3D structure aligns with observed shadows.
4. Silhouette Loss: Projects the predicted shape from five canonical viewpoints (top, 0°, 45°, 90°, 135°) and compares silhouettes using LPIPS to constrain the global crown outline.
5. Total Loss: A weighted sum of the above terms ($\mathcal{L}_{total} = \lambda_{occ}\mathcal{L}_{occ} + \lambda_{col}\mathcal{L}_{col} + \lambda_{sh}\mathcal{L}_{sh} + \lambda_{silh}\mathcal{L}_{silh}$).

D. Post-Processing

At inference, the model ranks query points by occupancy probability, retains the top-K points, and upsamples the sparse set via voxel-based resampling to create a dense, visually consistent point cloud.

3. Key Contributions

1. Synthetic Dataset: A large-scale dataset of 3,000 procedurally generated trees with aligned orthophotos, DSMs, and ground-truth point clouds.
2. Novel Architecture: A neural network capable of reconstructing detailed 3D tree point clouds from a single orthophoto and DSM, utilizing a PointNet/ResNet fusion approach.
3. Multi-Modal Supervision: A training strategy combining occupancy prediction with differentiable shadow and silhouette losses, enabling the model to infer 3D structure from 2.5D inputs without species priors.
4. Public Release: The codebase, synthetic dataset, and pretrained models are made publicly available.

4. Results and Evaluation

The method was evaluated on synthetic data, Austrian landmark trees (real-world), and French Pyrenees LiDAR data.

• Quantitative Performance:

  • Ablation Studies: Showed that combining DSM and Orthophoto inputs with all loss types (BCE + Shadow + Silhouette) yields the best results. Shadow/Silhouette losses alone are insufficient but significantly improve structure when combined with BCE.
  • Comparison with SOTA: TreeON outperformed baselines (including OpenLRM, Zero123++, AtlasNet, and diffusion models) across all metrics:
    • Chamfer Distance (CD): 0.969 (lowest).
    • F1-Score: 0.884 (highest, indicating high precision and recall).
    • Coverage (COV): 90.7% (indicating high diversity and lack of mode collapse).
  • Baselines often oversmoothed canopies, missed trunks, or produced noisy/fragmented shapes.

• Qualitative Performance:

  • Generalization: Successfully reconstructed diverse tree species (coniferous/deciduous) without explicit species labels.
  • Robustness: The dual-input design allowed the model to compensate for degraded inputs (e.g., if the DSM underestimated size, the orthophoto compensated, and vice versa).
  • Real-World Validation: Reconstructions of Austrian landmark trees matched morphological traits (height, crown size) observed in real photographs. Comparison with IGN LiDAR data confirmed alignment in height and crown spread.

• Efficiency:

  • Inference Speed: ~0.3 seconds per tree on a single GPU.
  • Storage: ~0.2 MB per tree (vs. tens of MBs for high-detail meshes).

5. Significance and Limitations

Significance:
TreeON bridges the gap between oversimplified 2.5D maps and computationally intensive 3D reconstructions. It enables the creation of scalable, interactive digital 3D maps for rural environments using only standard, widely available geospatial data (orthophotos and DSMs). It satisfies four critical requirements simultaneously: visual appeal, structural plausibility, computational efficiency, and full automation.

Limitations:

• Data Bias: The model relies on synthetic training data. It may struggle with rare species, unusual crown shapes, or seasonal variations (e.g., leafless winter trees or snow cover) not well-represented in the summer-foliage-heavy training set.
• Ground Truth Scarcity: Quantitative evaluation on real-world data is limited by the lack of high-quality paired 3D ground truth, relying heavily on qualitative assessment.
• Fine Details: While the overall structure is plausible, fine branching details may differ from high-resolution LiDAR scans.

In conclusion, TreeON represents a significant step forward in automated geospatial 3D modeling, offering a practical solution for generating realistic vegetation models at a landscape scale.