Test-Time Adaptation for Height Completion via Self-Supervised ViT Features and Monocular Foundation Models

1. Problem Statement

The paper addresses the challenge of Digital Surface Model (DSM) completion, specifically the reconstruction of accurate, metric height maps from incomplete or outdated prior data combined with up-to-date monocular (single-view) RGB imagery.

• Context: Large-scale DSMs often contain gaps due to sensor limitations, reconstruction artifacts, or changes in the built environment (e.g., new construction).
• Limitations of Existing Methods:
  • Traditional Interpolation: Methods like kriging or inverse distance weighting assume spatial continuity. They fail when entire objects (e.g., buildings) are missing from the prior, as they cannot infer the height of non-existent structures.
  • Supervised Learning: Deep learning approaches often require sensor-specific training, leading to poor generalization across domains and sensing conditions.
  • Monocular Height Estimation (MHE): While foundation models (e.g., Depth Anything V2) can estimate relative depth, they suffer from scale-height ambiguity in remote sensing due to near-orthographic viewing geometry (lack of perspective cues). They often lack metric accuracy without specific priors.
  • Prior-Based Rescaling: Existing methods that align relative depth to sparse priors often use global linear fitting or local weighted regression. These fail when objects are completely missing or when the prior is highly sparse, as they cannot transfer geometric information from semantically similar but spatially distant regions.

2. Methodology: Prior2DSM

The authors propose Prior2DSM, a training-free framework that operates entirely at Test-Time Adaptation (TTA). It leverages foundation models to propagate metric information from incomplete priors through semantic feature-space correspondence.

Core Components:

1. Monocular Foundation Models (Relative Depth):

   • The framework accepts an RGB image and uses a pre-trained monocular depth estimation (MDE) backbone (e.g., Depth Anything V2, Depth Pro, or MoGe-2) to generate a relative height map ($H_{rel}$).
   • This map captures structural geometry but lacks absolute metric scale.

2. Self-Supervised Semantic Features (DINOv3):

   • A DINOv3 Vision Transformer (ViT) encoder extracts dense semantic features ($F$) from the RGB image.
   • Dense Feature Extraction: To overcome the "blocky" nature of standard ViT patch grids, the authors implement a strided overlap accumulation strategy. By shifting the input image by a sub-patch stride, they average overlapping tokens to create high-resolution, dense semantic maps. This allows for fine-grained structural preservation.

3. Test-Time Adaptation (TTA) Mechanism:

   • Instead of training a new model, the system adapts the frozen foundation models during inference using the available (incomplete) metric priors ($H_{prior}$).
   • Low-Rank Adaptation (LoRA): LoRA is applied to the attention projection layers of the DINOv3 encoder. This allows the model to adapt its attention mechanisms to the specific scene without updating the full parameter set.
   • MLP Scale and Shift Predictor: A lightweight Multi-Layer Perceptron (MLP) takes the semantic token embeddings and predicts spatially varying scale ($s$) and shift ($b$) parameters.
   • Transformation: The relative depth is converted to metric height ($\hat{z}$) using the affine transformation: $\hat{z} = s \cdot r + b$.
   • Optimization: The MLP and LoRA parameters are optimized using the available metric pixels in the prior map (the "anchors") to minimize the error between the predicted metric height and the known prior heights. This enables the model to learn how to scale the relative depth locally based on semantic similarity.

4. Completion Strategy:

   • For regions where the prior is missing (e.g., new buildings), the model infers the appropriate scale and shift parameters by finding semantically analogous regions (e.g., other buildings of similar type) in the feature space, regardless of spatial distance. This allows for the reconstruction of fully missing objects.

3. Key Contributions

• Training-Free Framework: Introduces a novel approach for metric DSM completion that requires no task-specific supervised training, relying instead on foundation models and TTA.
• Semantic-Geometric Coupling: Demonstrates that self-supervised ViT features (DINOv3) encode a strong coupling between semantic similarity and geometric consistency, enabling height transfer across spatially separated but semantically similar objects.
• Robustness to Missing Data: Unlike interpolation or global rescaling, the method can complete fully missing objects (e.g., new construction) by leveraging semantic priors.
• Systematic Benchmarking: Provides a comprehensive evaluation of multiple state-of-the-art monocular foundation backbones (Depth Anything V2, Depth Pro, MoGe-2) within the TTA framework.
• Practical Applications: Validates the method on real-world DSM updating (2008–present temporal correction) and coupled RGB-DSM generation via integration with text-to-image models (Text2Earth).

4. Experimental Results

The method was evaluated on two high-resolution datasets: NAIP (Denver, USA; complex urban, high-rise) and WorldView-3 (Fresno, USA; residential, low-rise). Performance was tested under 25%, 50%, and 75% prior incompleteness.

• Quantitative Performance:
  • RMSE Reduction: Prior2DSM achieved up to a 46% reduction in RMSE compared to linear fitting of monocular depth models (e.g., 5.31m vs. 9.91m RMSE on NAIP at 25% incompleteness).
  • Comparison to SOTA: It consistently outperformed non-parametric baselines (bilinear interpolation, LWLR, Global Rescaling) and state-of-the-art prior-based depth completion models (PriorDA, Marigold-DC).
  • Structural Fidelity: Achieved higher SSIM scores, indicating better preservation of structural details and less smoothing of edges compared to interpolation methods.
• Ablation Studies:
  • LoRA: Removing LoRA (using a frozen DINOv3) increased RMSE, confirming the necessity of adapting attention mechanisms to the specific scene.
  • Backbone: While all tested backbones worked, MoGe-2 and Depth Pro generally provided slightly better geometric consistency than Depth Anything V2, though the framework remained robust across all choices.
• Real-World Application:
  • DSM Updating: Successfully updated a 2008 DSM with 2024 imagery, reducing RMSE from 5.35m to 3.83m.
  • Co-Editing: Demonstrated the ability to generate metrically consistent DSMs for newly generated buildings in RGB images using generative AI.

5. Significance and Conclusion

This work bridges the gap between semantic understanding and metric geometric reconstruction in remote sensing. By utilizing Test-Time Adaptation on foundation models, Prior2DSM overcomes the limitations of domain-specific training and spatial interpolation.

• Impact: It enables the creation of accurate, up-to-date 3D maps from legacy or incomplete data sources, which is critical for urban planning, disaster response, and infrastructure management.
• Innovation: It shifts the paradigm from "training a model for a specific sensor" to "adapting general foundation models at inference time," offering a scalable, sensor-agnostic, and object-agnostic solution for height completion.
• Limitations: The method relies on the quality of the initial relative depth estimation and incurs a computational cost during the TTA optimization phase (approx. 8.2 mins/km²), though this is acceptable for localized updates.

In summary, Prior2DSM represents a significant advancement in remote sensing by demonstrating that semantic feature spaces can effectively guide the metric completion of 3D surfaces, even when large portions of the prior data are missing.