APCoTTA: Continual Test-Time Adaptation for Semantic Segmentation of Airborne LiDAR Point Clouds

This paper proposes APCoTTA, a novel Continual Test-Time Adaptation framework for airborne LiDAR semantic segmentation that combines gradient-driven layer selection, entropy-based consistency regularization, and random parameter interpolation to mitigate catastrophic forgetting and error accumulation, while also introducing two new benchmarks to address the lack of evaluation standards in this domain.

The Gist

The Big Picture: The "Smart Map" That Gets Lost

Imagine you have a very smart robot named LiDAR. Its job is to look at the world from a helicopter or drone and draw a perfect 3D map, labeling everything it sees: "That's a tree," "That's a road," "That's a house."

When LiDAR is first built, it is trained in a perfect, controlled environment (let's call it School City). It learns to recognize trees and roads perfectly there.

The Problem:
Now, you send LiDAR out into the real world. But the real world is messy!

• The Weather Changes: One day it's sunny, the next it's foggy. The sun might blind the sensors.
• The Terrain Changes: You fly from a dense city to a quiet village. The trees look different; the houses are built differently.
• The Hardware Ages: The sensor gets a little dusty or vibrates differently over time.

Because of these changes, the data LiDAR sees looks nothing like the "School City" data it was trained on. If LiDAR keeps using its old "School City" brain, it starts making mistakes. It might think a cloud is a roof, or a shadow is a car.

The Old Solutions (and why they failed):

1. Retraining: You could send LiDAR back to school to relearn everything. But this is too expensive and slow. You can't stop the helicopter every time the weather changes.
2. Standard Adaptation: Some methods try to tweak the brain slightly while flying. But they often get confused by the noise, forget what they learned in school, and start hallucinating (calling a bush a car). This is called "Catastrophic Forgetting."

The New Solution: APCoTTA

The authors created a new system called APCoTTA (Airborne Point cloud Continual Test-Time Adaptation). Think of APCoTTA as a Smart Pilot that helps LiDAR adapt on the fly without losing its mind.

Here is how APCoTTA works, broken down into three simple tricks:

1. The "Selective Brain Surgeon" (Dynamic Layer Selection)

• The Analogy: Imagine LiDAR's brain is a huge library with thousands of books. Some books are about "General Rules" (like "trees are green"), and some are about "Specific Details" (like "how this specific type of roof looks").
• The Problem: When the environment changes, you don't want to rewrite the "General Rules" books, or you'll forget the basics. But you do need to update the "Specific Details" books.
• The APCoTTA Fix: APCoTTA acts like a surgeon. It checks which parts of the brain are "confused" (low confidence). It freezes the stable parts (the General Rules) so they don't get corrupted. It only allows the confused parts to learn and update. This way, LiDAR adapts to the new scenery without forgetting how to recognize a tree.

2. The "Trustworthy Filter" (Entropy-Based Consistency)

• The Analogy: Imagine LiDAR is looking at a foggy window. It tries to guess what's behind it. Sometimes it's very sure ("It's a car!"), and sometimes it's just guessing wildly ("Maybe it's a ghost?").
• The Problem: If you teach LiDAR based on its wild guesses, it will learn the wrong things. This is called "Error Accumulation."
• The APCoTTA Fix: APCoTTA has a filter. It looks at LiDAR's guesses and asks, "Are you sure?" If LiDAR is shaky and unsure (high entropy), APCoTTA says, "Ignore that guess, it's probably wrong." It only lets LiDAR learn from the guesses where it is highly confident. This stops the robot from learning from its own mistakes.

3. The "Safety Net" (Randomized Parameter Interpolation)

• The Analogy: Imagine LiDAR is walking on a tightrope. If it leans too far toward the new environment (the "Target"), it might fall off the rope and forget everything about the old environment (the "Source").
• The Problem: If LiDAR adapts too aggressively, it forgets its original training. If it adapts too little, it can't handle the new weather.
• The APCoTTA Fix: APCoTTA uses a "Safety Net." Every now and then, it gently pulls LiDAR's brain back toward its original "School City" settings. It doesn't do a hard reset (which would wipe out all progress); instead, it blends the new knowledge with a tiny bit of the old knowledge. This keeps LiDAR balanced, allowing it to adapt to the new world without forgetting who it is.

The Result: A New Benchmark

The authors realized that nobody had a standard "test track" for this kind of problem. So, they built two new test tracks (called ISPRSC and H3DC) that simulate bad weather, sensor errors, and changing landscapes.

They tested APCoTTA on these tracks and found:

• Old methods struggled, often getting confused and forgetting what they knew.
• APCoTTA was a champion. It improved accuracy by 9% to 14% compared to just using the raw model.

Summary

APCoTTA is like giving a robot a smart pilot that:

1. Only updates the parts of the brain that need it (saving the rest).
2. Ignores the robot's bad guesses to prevent learning errors.
3. Gently nudges the robot back to its roots so it doesn't forget its training.

This allows airborne LiDAR systems to fly anywhere, in any weather, and keep mapping the world accurately without needing to stop and retrain.

Technical Summary

1. Problem Statement

Airborne LiDAR (ALS) point cloud semantic segmentation is critical for large-scale 3D scene understanding (e.g., mapping, forest monitoring). However, models trained on static source domains often suffer severe performance degradation when deployed in real-world scenarios due to continuous domain shifts. These shifts arise from:

• Spatiotemporal Variability: Changes in terrain (urban to rural), seasons, and vegetation phenology.
• Sensor and Environmental Factors: Variations in flight altitude, sensor degradation, weather conditions (sunlight, rain), and noise.
• Limitations of Existing Methods:
  • Retraining: Requires expensive, dense 3D annotations.
  • Unsupervised Domain Adaptation (UDA): Requires access to source data, which is often restricted by privacy.
  • Standard Test-Time Adaptation (TTA): Assumes a static target domain. In ALS, continuous shifts lead to error accumulation (from unreliable pseudo-labels) and catastrophic forgetting (loss of source knowledge).
  • Existing CTTA: Most Continual Test-Time Adaptation (CTTA) methods are designed for 2D images. Directly applying them to unstructured, sparse, and geometrically complex ALS point clouds results in unstable adaptation and failure to preserve geometric priors.

Core Challenge: There is a lack of a standardized benchmark for ALS CTTA, and existing methods fail to balance target adaptation with source knowledge retention in the presence of continuous, evolving distributions.

2. Methodology: APCoTTA

The authors propose APCoTTA, a novel CTTA framework tailored specifically for ALS point clouds. It operates without access to source data and consists of three synergistic modules:

A. Dynamic Selection of Trainable Layers (DSTL)

• Goal: Mitigate catastrophic forgetting by preventing the model from overwriting stable source knowledge.
• Mechanism: Instead of updating all parameters (full-parameter update), the method selectively updates only specific layers.
  • It computes the Kullback-Leibler (KL) divergence between the model's predictions on weakly augmented views and a uniform distribution.
  • Gradient Norms: Layers with low gradient norms (indicating high confidence/familiarity with the data) are frozen. Layers with high gradient norms (indicating uncertainty or sensitivity to geometric shifts) are marked as trainable.
  • This ensures the model adapts to new geometric structures while preserving robust source domain priors.

B. Entropy-Based Consistency Loss (EBCL)

• Goal: Reduce error accumulation caused by incorrect pseudo-labels from low-confidence samples.
• Mechanism:
  • The model processes both weakly ($X^w_t$) and strongly ($X^s_t$) augmented views.
  • Entropy Filtering: The Shannon entropy of predictions on the weak view is calculated. Samples with entropy above a fixed global threshold $\tau$ (indicating extreme uncertainty) are discarded.
  • Consistency: The consistency loss is computed only on the remaining high-confidence samples, ensuring the model learns from reliable data and avoids propagating noise.

C. Randomized Parameter Interpolation (RPI)

• Goal: Provide a "soft" regularization anchor to prevent drift from the source domain without abrupt resets.
• Mechanism:
  • Unlike prior works that use "hard" stochastic resets (randomly resetting weights to source values), APCoTTA employs a soft fusion strategy.
  • For a subset of trainable parameters (selected with probability $p$), the updated parameters are interpolated with the original source parameters:
    $$\theta_{t+1} = M \odot (\alpha \theta_0 + (1-\alpha)\tilde{\theta}_t) + (1-M) \odot \tilde{\theta}_t$$
  • This balances adaptation to the target domain with retention of source knowledge, smoothing the regularization trajectory.

3. Key Contributions

1. New Benchmarks (ISPRSC & H3DC):

   • The authors constructed two robust benchmark datasets based on the ISPRS and H3D datasets.
   • They simulate 7 types of corruptions (e.g., strong sunlight, density decrease, cutout, impulse noise, Gaussian noise) across 5 severity levels to mimic real-world ALS challenges.
   • These fill the gap in standardized evaluation protocols for ALS CTTA.

2. APCoTTA Framework:

   • A specialized CTTA framework that addresses the unique challenges of unstructured point cloud data (irregular sparsity, geometric instability).
   • It explicitly handles catastrophic forgetting (via DSTL and RPI) and error accumulation (via EBCL).

3. Synergistic Modules:

   • DSTL: Uses gradient norms to identify and freeze stable layers.
   • EBCL: Filters out low-confidence samples to ensure stable learning.
   • RPI: Uses soft parameter interpolation to maintain source alignment.

4. Comprehensive Evaluation:

   • Extensive experiments demonstrating superior performance over state-of-the-art baselines (including CoTTA, PALM, and Wang et al.) on both benchmarks.

4. Experimental Results

The method was evaluated on two tasks: ISPRS $\to$ ISPRSC and H3D $\to$ H3DC.

• Performance Gains:
  • ISPRSC: APCoTTA achieved an mIoU of 49.74%, an improvement of ~9% over direct inference (Source) and outperformed the next best method (Wang et al.) by ~2.5%.
  • H3DC: APCoTTA achieved an mIoU of 46.22%, an improvement of ~14% over direct inference (Source) and significantly outperformed all baselines.
• Comparison with Baselines:
  • Standard TTA methods (TENT, Pseudo-label) suffered from severe error accumulation and catastrophic forgetting, leading to performance drops over time.
  • Image-based CTTA methods (CoTTA, PALM) struggled with the geometric instability of point clouds.
  • APCoTTA maintained high accuracy across all corruption types, demonstrating robustness to impulse noise, Gaussian noise, and density variations.
• Ablation Studies:
  • Removing any single module (DSTL, EBCL, or RPI) resulted in significant performance degradation, confirming that all three components are necessary for optimal adaptation.
  • Hyperparameter sensitivity analysis showed the model is robust within specific ranges but peaks at optimal settings (e.g., sampling probability $p=0.01$).

5. Significance

• Practical Deployment: APCoTTA enables the deployment of ALS segmentation models in dynamic, real-world environments where data distributions continuously evolve, without requiring retraining or access to source data.
• Field Advancement: By introducing the first dedicated CTTA benchmarks for airborne LiDAR, the paper establishes a standard for evaluating robustness against environmental and sensor corruptions.
• Methodological Insight: The work highlights that image-based CTTA assumptions (e.g., stable feature statistics, simple augmentations) do not hold for point clouds. It proposes a new paradigm for 3D adaptation that prioritizes geometric stability and selective parameter updates.

Limitations & Future Work:
The authors note that class imbalance (long-tail classes) is not fully addressed and plan to extend the framework to open-set scenarios to handle unknown classes appearing during deployment.