💻 computer science

Depth Estimators Are Implicit Neural Fields for 3D Scene Geometry Inpainting and Reconstruction

This paper proposes Neural Depth Field (NDF), a novel approach that reinterprets depth estimators as scene-level implicit fields to perform single test-time optimization, thereby achieving state-of-the-art 3D geometry inpainting and reconstruction with high fidelity and global consistency across diverse datasets.

Original authors: Yingzhao Jian, Zihao Lin, Hehe Fan

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yingzhao Jian, Zihao Lin, Hehe Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of the world, like a video game level that looks exactly like real life. To do this, computers need to know the shape of everything: where the walls are, how high the mountains stand, and how deep the holes go. This is called "3D geometry." Usually, computers get this information from cameras or sensors, but real life is messy. Sometimes clouds block the view, sometimes shadows hide details, and sometimes the sensors just miss a spot, leaving a "hole" in the data. It's like trying to finish a jigsaw puzzle when someone has eaten a few pieces; you can see the picture, but there are gaps where the image is missing.

To fix these holes, scientists often use "depth estimators." Think of these as super-smart guessers. They have studied millions of photos and learned what a wall, a tree, or a building usually looks like. When they see a gap, they use that knowledge to fill it in. However, these guessers have two big problems. First, they sometimes guess things that don't fit with the parts of the picture they can actually see (like guessing a door is in a spot where there is clearly a solid wall). Second, if they see something they've never seen before—like a view from a satellite high above the city instead of a street-level view—they might get confused and make wild, wrong guesses.

This paper introduces a clever new way to fix these problems called Neural Depth Field (NDF). The researchers realized that a depth estimator doesn't have to be just a "one-shot guesser." Instead, they treated it like a flexible, invisible mold that can be reshaped to fit the specific scene it is looking at. Imagine you have a lump of clay that knows how to make a perfect cup. Usually, you just press it once and hope it looks right. But with NDF, you press the clay against the actual cup you are holding, smoothing it out until it matches the real curves perfectly, while still keeping the general shape of a cup. This allows the computer to fill in the missing holes with high precision, making sure the new parts blend seamlessly with the old parts, whether it's a messy indoor room or a giant city seen from space.

The Problem with "One-Shot" Guessing

In the world of 3D computer vision, we often rely on pre-trained models to tell us how deep things are. These models are like students who have studied a massive textbook of 3D shapes. When you show them a picture, they instantly guess the depth. But real-world data is often incomplete. A drone might miss a building because of a tree, or a satellite might lose data because of a cloud.

The standard approach is to take these pre-trained models and ask them to "inpaint" (fill in) the missing parts. However, the authors point out that this often fails in two specific ways:

Inconsistency: The model might fill in a hole with a shape that contradicts the visible parts of the scene. It's like drawing a door on a wall that the rest of the photo clearly shows is a solid brick surface.
Out-of-Distribution Confusion: If the model was trained mostly on street-level photos, it might get totally lost when looking at a satellite image. It doesn't know the rules for that specific "domain," so its guesses become unreliable.

The "Neural Depth Field" Solution

The authors propose a new method called Neural Depth Field (NDF). Their big idea is to stop treating the depth estimator as a static tool that just gives an answer, and start treating it as a scene-level implicit field.

Here is the analogy: Imagine a depth estimator is a rubber sheet with a pattern printed on it.

As a Predictor: Normally, you just stretch this sheet over a picture, and it prints a guess based on its pattern.
As a Field: With NDF, you take that same rubber sheet and physically press it against the actual parts of the picture you can see. You stretch and smooth the sheet until it hugs the real, visible geometry perfectly.

By doing this "test-time optimization" (adjusting the model while it's looking at the specific scene), NDF forces the model to do two things at once:

Fit the Observed: It must match the parts of the image we can actually see, ensuring the new guesses don't contradict the old data.
Adapt the Prior: It uses its general knowledge (the "pattern" on the sheet) to guess what the missing parts should look like, but now that knowledge is tuned specifically for this scene.

This turns the problem of "filling in the blanks" into a single, unified task: finding the best version of the rubber sheet that fits both the known parts and the unknown parts simultaneously.

What They Found

The researchers tested this idea on two very different types of scenes:

Satellite Imagery: They used it on a massive city map (Birmingham) where the height data was missing in many spots due to clouds or sensor errors.
Indoor Scans: They used it on 3D scans of rooms (from the ScanNet dataset) where furniture and walls were missing due to occlusions or sensor noise.

The Results:

Better Accuracy: NDF filled in the missing parts much more accurately than previous methods. On the satellite data, it improved the accuracy of filling in missing height by 23.1%.
Fewer Glitches: It fixed the problem of the new parts clashing with the old parts. It reduced "cross-view inconsistency" (where the picture looks different or broken when you switch angles) by 63.3%.
Fine Details: The method was able to recover tiny details that other methods missed, like small streetlamps in satellite photos or thin wires in indoor scans.

The paper explicitly rules out the idea that simply using a pre-trained model without adjustment is enough. They showed that just "predicting" or just "reconstructing" separately leads to blurry or inconsistent results. The magic happens only when you combine the two into this single optimization process.

Why It Matters

This isn't just about making prettier pictures. Having a complete, consistent 3D map of the world is crucial for robots to navigate, for cities to plan infrastructure, and for virtual reality to feel real. If a robot thinks a wall is there when it isn't, it might crash. If a city planner thinks a building is a certain height when it's actually taller, it could be a safety hazard.

By showing that we can take a general "guessing" model and tune it to fit a specific scene perfectly, NDF offers a way to get high-quality 3D data even when our sensors fail or when we are looking at things from angles we've never seen before. It suggests that the future of 3D reconstruction isn't just about building bigger, smarter models, but about teaching those models how to adapt to the specific reality they are looking at, right then and there.

The authors note that this process currently takes a few minutes to an hour per scene, which is slower than a simple "one-shot" guess, but the trade-off is a much higher quality result. They suggest that in the future, this could be sped up, but for now, it proves that taking the time to "fit the mold" to the scene is worth it for getting a truly accurate 3D world.

Technical Summary: Neural Depth Field (NDF) for 3D Scene Geometry Inpainting and Reconstruction

1. Problem Definition

Real-world 3D scene data is frequently structurally incomplete due to occlusions, sensor limitations, or environmental factors (e.g., clouds in satellite imagery, motion blur in indoor scans). While mainstream depth estimators can recover continuous geometry from partial observations, they face two critical limitations:

Inconsistency: The inpainted regions often fail to align geometrically with the observed parts, leading to discontinuities.
Distribution Shift: Standard depth estimators, trained on large-scale datasets, often perform unreliably on out-of-distribution data (e.g., satellite imagery or specific indoor textures) and struggle to maintain metric scale consistency across different views.

Existing approaches either rely on direct prior extrapolation (which is unreliable for unseen data) or use depth estimators as indirect supervision for neural fields (which can lead to inconsistencies between the predicted depth and the optimized scene field).

2. Methodology: Neural Depth Field (NDF)

The authors propose Neural Depth Field (NDF), a test-time optimization framework that reinterprets a pretrained depth estimator as a scene-level implicit neural field.

Core Insight

NDF operates on the dual perspective that a depth estimator serves two simultaneous roles:

As a Predictor: It learns domain-specific priors for depth inpainting by optimizing on existing depth observations.
As an Implicit Field: It fits the observed geometry to maintain global consistency.

Formulation

Given an observation $O = (I, D, M)$ , where $I$ is the RGB image, $D$ is the incomplete depth map, and $M$ is a validity mask, the goal is to find an optimal depth estimator $f^*$ that minimizes the gap between the predicted depth $\tilde{D}$ and the complete ground truth $\hat{D}$ .

NDF treats the parameters $\theta$ of a pretrained mono-depth estimator $f_\theta$ as the variables of a scene-level implicit field $F_\theta$ . The optimization objective unifies two tasks into a single pixel-wise discrepancy loss:

Geometry Consistency (Fitting): The rendered geometry must fit the observed depth $D|_{\Omega_v}$ (where $M=1$ ).
Domain Adaptation (Inpainting): The prediction must be supervised by the ground-truth depth distribution to adapt the prior to the current scene, while filling in the hidden regions $\Omega_h$ (where $M=0$ ).

The unified loss function is:
$\ell(\theta) = \int_{\Omega} M(u)\rho(F_\theta(u), D(u)) d\nu(u) + \alpha \Pi_{F_\theta}$
Where the first term ensures consistency with observed data, and $\Pi_{F_\theta}$ represents constraints such as pretrained priors or smoothness regularizers. By optimizing this shared loss at test time, NDF performs inpainting in missing regions while simultaneously reconstructing and refining observed regions.

Implementation Details

Base Model: NDF is instantiated using InfiniDepth, a state-of-the-art monocular depth estimator.
Optimization: The model undergoes test-time optimization on specific scenes (e.g., satellite windows or indoor frames).
Augmentations: The framework employs transferred-mask augmentation (randomly transferring invalid masks between windows) and overlap-window losses to enforce consistency across sliding windows.
Indoor Specifics: For indoor scans, a surface normal loss is added to align with a teacher model, mitigating noise from scan artifacts.

3. Key Contributions

Unified Formulation: The paper is the first to systematically formulate depth estimators as implicit neural fields, unifying 3D scene inpainting and reconstruction into a single test-time optimization process.
Dual-Role Mechanism: It demonstrates that treating a depth estimator as both a predictor and an implicit field allows for simultaneous domain adaptation and geometric consistency.
Scalable Pipeline: The authors provide a scalable pipeline for high-fidelity scene data inpainting and generation, including processed datasets with completed geometry to support future research.

4. Experimental Results

The method was evaluated on two distinct datasets: Birmingham (city-scale satellite imagery) and ScanNet (indoor 3D scans).

Quantitative Performance

Satellite Imagery (Birmingham):
- Inpainting Accuracy: NDF improved metric-depth inpainting accuracy by 23.1% and relative-depth accuracy by 9.2% compared to the baseline InfiniDepth.
- Consistency: It reduced cross-view inconsistency (Overlap MAE) by 63.3% and improved regression accuracy on observed regions (Observed MAE) by 55.4%.
Indoor Scans (ScanNet):
- Inpainting Accuracy: Achieved a 13.2% reduction in AbsRel for metric-depth and a 17.4% gain for relative-depth in held-out regions.
- Consistency: Reduced Overlap MAE by 32.5% (metric) and 60.3% (relative), significantly outperforming methods that treat prediction and reconstruction separately.

Qualitative Performance

Visual comparisons show that NDF successfully recovers fine-grained structures (e.g., streetlamps in satellite images, thin wires in indoor scans) and maintains plausible metric scales. Unlike standard predictors that hallucinate structures or produce blurry reconstructions, NDF preserves local geometric coherence and global cross-view consistency, seamlessly blending inpainted regions with observed geometry.

5. Significance and Claims

The paper claims that NDF represents a significant shift in how depth estimators are utilized for 3D reconstruction. By moving from a "one-shot prediction" paradigm to a "test-time optimization" paradigm, NDF addresses the fundamental trade-off between leveraging pretrained priors and maintaining scene-specific consistency.

The authors emphasize that their approach:

Solves the inconsistency problem without requiring complex multi-view geometry optimization from scratch.
Handles out-of-distribution data (like satellite imagery) by adapting the prior to the specific scene during inference.
Achieves state-of-the-art performance in both accuracy and global consistency across diverse scene types.

Limitations and Future Work:
The authors modestly acknowledge that NDF currently requires per-scene test-time optimization (taking minutes to an hour), which limits real-time application. They suggest that amortizing this cost via meta-learning or parameter-efficient adaptation is a natural next step. Additionally, the current formulation targets static scenes; extending the method to dynamic scenes and richer representations (e.g., Gaussian Splatting) is identified as future work.