Inpainting the Red Planet: Diffusion Models for the Reconstruction of Martian Environments in Virtual Reality

This paper proposes an unconditional diffusion model trained on augmented HiRISE heightmaps to reconstruct missing Martian terrain data in virtual reality, demonstrating superior accuracy and perceptual similarity compared to traditional interpolation methods.

The Gist

🚀 The Big Picture: Fixing the "Glitchy" Map of Mars

Imagine you are trying to build a hyper-realistic Virtual Reality (VR) simulation of Mars for astronauts to train in. You need a perfect 3D map of the surface. But here's the problem: The cameras on our satellites orbiting Mars are like old, dusty binoculars. Sometimes they miss a spot, sometimes the signal gets lost in space, and sometimes the data just doesn't make it back to Earth.

The result? Your 3D map of Mars has giant holes in it. It's like a jigsaw puzzle where someone ripped out the middle pieces. If you try to walk through this VR world, you'd fall into a black void.

The Goal: The authors wanted to fill in those missing holes so the map looks smooth, realistic, and safe for astronauts to explore.

---

🧩 The Old Way: "Guessing" the Missing Pieces

Before this paper, scientists used two main ways to fix these holes:

1. The "Neighborhood Watch" (Interpolation): Imagine you are missing a tile in a floor. You look at the tiles immediately next to the hole and guess the color based on them. If the neighbors are blue, you paint the hole blue.
   • The Problem: This is too simple. Mars isn't just flat blue; it has craters, dunes, and ridges. If you just "average" the neighbors, you end up with a flat, blurry blob that looks nothing like a real Martian landscape.
2. The "Fluid Flow" (Navier-Stokes): This treats the missing area like a puddle of water. It tries to "flow" the surrounding terrain into the hole to smooth it out.
   • The Problem: While better than the first method, it still struggles to create complex features like sharp crater edges or sand dunes. It tends to make things look too smooth and artificial.

---

🎨 The New Way: The "AI Artist" with a Memory

The authors decided to use a Diffusion Model. Think of this not as a calculator, but as a super-creative artist who has studied thousands of photos of Mars.

Here is how their "AI Artist" works:

1. The Training (Learning the Vibe): They fed the AI 12,000 pictures of Martian terrain. They didn't just show it one big map; they chopped them up into tiny, random pieces of different sizes. This taught the AI that Mars has big features (like giant craters) and small features (like tiny rocks), all mixed together.
2. The Magic Trick (The Diffusion Process): Imagine the AI starts with a picture that is completely covered in static (like an old TV with no signal).
   • The AI knows what a "clean" Mars looks like.
   • It slowly removes the static, pixel by pixel, asking itself: "If I see a crater here, what should the pixels around it look like?"
3. The "Unconditional" Superpower: Most AI art tools need a prompt, like "Draw a red rock." But for Mars, we often don't have that extra info. The authors built an Unconditional model. This means the AI doesn't need a prompt. It just looks at the edges of the hole and says, "I know what a crater looks like. I know what a dune looks like. I will just paint the missing part to fit perfectly."

The Analogy:

• Old Methods: Like trying to fix a torn photo by taping it together with clear tape. You can see the tear, and the image is blurry.
• This New Method: Like hiring a master painter who has memorized the entire photo. They look at the tear, remember what the scene should look like, and paint over the tear so seamlessly that you can't tell it was ever damaged.

---

📊 Did It Work? (The Results)

The team tested their "AI Artist" against the old methods (the "Neighborhood Watch" and "Fluid Flow") on 1,000 different maps.

• The Visual Test: They put the maps into a 3D VR engine.
  • Old methods: Looked like plastic or smooth clay. The craters looked fake.
  • New method: Looked like real Mars. The craters had depth, the rocks looked sharp, and the transition from the "real" data to the "filled-in" data was invisible.
• The Math Test: They measured the errors.
  • The new method was 4% to 15% more accurate in height measurements than the old methods.
  • It was 29% to 81% more similar to the original data in terms of how the human eye perceives the image.

🌍 Why Does This Matter?

This isn't just about making pretty pictures.

• Astronaut Training: If an astronaut is training in VR to drive a rover, they need to see a real crater. If the AI fills that crater with a flat blob, the astronaut might crash in real life.
• Mission Planning: Scientists need to know exactly how high a hill is to plan a landing.
• Data Scarcity: Mars is far away. We can't just go back and take a new photo if we miss a spot. This AI allows us to "hallucinate" (in a good way) the missing data based on what we already know about the planet.

🏁 The Bottom Line

The authors created a tool that acts like a digital time-traveling artist. It looks at the broken, incomplete maps of Mars we have, remembers what the planet looks like from its training, and fills in the missing pieces with such high quality that it creates a perfect, smooth, and realistic 3D world for us to explore virtually.

They proved that even when we don't have all the data, a smart AI can figure out the rest, making our virtual trips to Mars safer and more real than ever before.

Technical Summary

1. Problem Statement

Virtual Reality (VR) is increasingly critical for space exploration tasks, including mission planning, scientific analysis, and astronaut training. These applications rely on accurate 3D representations of planetary terrains, typically stored as heightmaps (raster grids encoding altitude).

However, extraterrestrial heightmaps derived from satellite imagery (e.g., NASA's HiRISE on the Mars Reconnaissance Orbiter) frequently suffer from missing values (voids) due to:

• Acquisition errors and signal degradation.
• Data loss during transmission.
• Processing ambiguities when converting images to heightmaps.

Existing solutions for filling these voids face significant limitations:

• Traditional Interpolation: Methods like Inverse Distance Weighting (IDW) and Kriging often fail to preserve complex geometric coherence and high-frequency terrain features (e.g., crater ridges, dunes).
• Conditional Deep Learning: Current deep learning approaches (GANs, conditional Diffusion Models) rely on auxiliary data (e.g., binary masks, terrain structure, or textual prompts) to guide generation. This is unsuitable for Mars because such auxiliary data is often unavailable, and models trained on Earth data tend to overfit to terrestrial geometries, failing to generalize to extraterrestrial surfaces.
• Visual Fidelity: Many existing methods prioritize quantitative error metrics over visual realism, which is crucial for immersive VR applications.

2. Methodology

The authors propose a novel framework using Unconditional Diffusion Models to reconstruct Martian surfaces without requiring auxiliary conditioning data.

A. Dataset Construction

• Source: NASA's HiRISE survey (Mars Reconnaissance Orbiter), providing high-resolution Digital Elevation Models (DEMs).
• Augmentation Strategy: Since raw HiRISE DEMs are large ($10^3$ pixels) and in a specialized PDS format, the authors created a custom dataset of 12,000 normalized heightmaps.
  • Random crops were extracted with side lengths between 512 and 2048 pixels.
  • Non-homogeneous Rescaling: Crops were resized to a fixed 128 × 128 resolution using nearest-neighbor interpolation. This strategy forces the model to learn terrain features across multiple scales, compensating for the relatively small dataset size and enhancing generalization.

B. Model Architecture & Training

• Model: A Denoising Diffusion Probabilistic Model (DDPM) using a U-Net architecture.
• Unconditional Approach: Unlike standard inpainting diffusion models that condition on a mask, this model is trained unconditionally on clean Martian terrain data. It learns the underlying distribution of Martian morphologies.
• Training Details:
  • 100 epochs, batch size of 16, 1000 timesteps.
  • Hardware: 2 NVIDIA A40 GPUs (~8 hours total training time).
  • Framework: Huggingface Diffusers library.

C. Inference (RePaint Algorithm)

To perform inpainting, the authors utilize the RePaint algorithm:

1. Input: A degraded heightmap (128 × 128) and a binary mask indicating missing regions.
2. Process: The model performs an unconditional reverse diffusion process.
3. Mask Guidance: At each denoising step, the valid (non-missing) pixels from the original input are overwritten onto the generated intermediate result. This guides the diffusion process to remain coherent with the known data while generating realistic content for the missing areas.
4. Resampling: A resampling mechanism adds and removes noise multiple times to ensure the generated patches are consistent with the surrounding valid pixels.

3. Key Contributions

1. Unconditional Reconstruction Framework: The first application of unconditional diffusion models for planetary terrain inpainting, eliminating the need for unavailable auxiliary conditioning data.
2. Augmented Multi-Scale Dataset: A curated dataset of 12,000 Martian heightmaps utilizing a non-homogeneous rescaling strategy to capture multi-scale terrain features.
3. Comprehensive Evaluation: A rigorous comparison against traditional void-filling (IDW, Kriging) and image inpainting (Navier-Stokes) methods using both quantitative metrics and 3D VR visualization.
4. Performance Gains: Demonstrated significant improvements in reconstruction accuracy and perceptual similarity, specifically tailored for VR applications.

4. Experimental Results

The method was evaluated on 1,000 random samples with artificially generated missing data (simulating aliasing artifacts).

A. Quantitative Metrics

The proposed method outperformed all baselines (IDW, Kriging, Navier-Stokes) across most error metrics:

• RMSE (Root Mean Squared Error): Improved by 4–15% compared to baselines.
• LPIPS (Learned Perceptual Image Patch Similarity): Improved by 29–81%, indicating significantly higher perceptual similarity to the original terrain.
• Other Metrics: Superior performance in MAE, PSNR, and EMD (Earth Mover's Distance).
• Note: Kriging achieved a slightly higher SSIM score, but the authors argue this is due to SSIM's sensitivity to structural alignment rather than perceptual realism.

B. Qualitative & Visual Assessment

• 3D Visualization: Using Autodesk Maya and the Arnold renderer, the authors visualized the reconstructed meshes.
• Findings:
  • IDW & Navier-Stokes: Produced discontinuities and "smoothing" artifacts, failing to reconstruct high-frequency features like crater ridges.
  • Kriging: Performed well on flat regions but struggled with complex structures, often producing artifacts that disrupted the overall surface structure.
  • Diffusion Model: Generated seamless transitions between valid and missing regions, preserving complex morphological patterns (dunes, craters) and producing geometrically consistent meshes suitable for VR.

5. Significance and Future Work

• VR Readiness: The study validates that diffusion models can generate terrain data that is not just mathematically accurate but visually and structurally coherent for immersive environments, a critical requirement for astronaut training and mission planning.
• Data Scarcity Solution: By using unconditional models, the approach overcomes the "data starvation" problem inherent in extraterrestrial research, where large, labeled, or conditioned datasets do not exist.
• Future Directions:
  • Conducting user studies to evaluate the "Sense of Presence" in VR.
  • Implementing multi-resolution models (e.g., Latent Diffusion Models, Diffusion Transformers) for faster inference.
  • Adapting the framework to other celestial bodies (e.g., the Moon).

Conclusion: The paper successfully demonstrates that unconditional diffusion models, trained on augmented multi-scale Martian data, provide a superior alternative to traditional interpolation for reconstructing degraded planetary heightmaps, enabling high-fidelity 3D visualization for space exploration.