Learning Scene-Level Signed Directional Distance Function with Ellipsoidal Priors and Neural Residuals

1. Problem Statement

The paper addresses the challenges in 3D scene representation for dense reconstruction and differentiable rendering. Existing methods face a trade-off between geometric accuracy, rendering efficiency, and differentiability:

• Explicit Representations (Meshes, Point Clouds, Voxels): Discontinuous and non-differentiable, hindering end-to-end optimization and novel view synthesis.
• Implicit Representations (NeRF, SDF):
  • NeRF/Gaussian Splatting: Excellent for photometric rendering but often struggle with accurate geometric reconstruction without careful supervision. They typically require integrating along rays, which is computationally expensive.
  • SDF (Signed Distance Function): Provides accurate geometry but requires iterative sphere tracing to find surface intersections along a specific direction. This process is slow, accumulates errors, and is not inherently differentiable in a single forward pass.
• The Gap: There is a need for a representation that supports single-query, differentiable directional distance prediction (like NeRF's input structure) while providing direct distance values (like SDF) to enable efficient geometric reconstruction and view optimization.

2. Methodology: Signed Directional Distance Function (SDDF)

The authors propose a novel representation called Signed Directional Distance Function (SDDF) and a hybrid learning framework to model it.

A. Definition of SDDF

Unlike standard SDF (which finds the shortest distance to any surface point) or Directional Distance Functions (DDF) that often lack sign information, the proposed SDDF $f(p, v)$ takes a position $p$ and a viewing direction $v$ as input.

• Output: The signed distance from $p$ to the first surface intersection along ray $v$.
• Sign Convention: Positive if $p$ is in free space, negative if $p$ is inside an object.
• Key Property: It satisfies a Directional Eikonal Equation: $v^\top \nabla_p f(p, v) = -1$. This means the distance decreases at a unit rate as one moves along the ray toward the surface. This property allows the model to be regularized without explicit loss terms.

B. Hybrid Explicit-Implicit Architecture

To learn scene-level SDDF efficiently and handle discontinuities (occlusions), the authors design a two-stage hybrid model:

1. Explicit Ellipsoid-Based Prior Network ($P$):

   • Purpose: Captures the coarse structure of the scene and handles occlusions/discontinuities.
   • Mechanism: The scene is approximated by a set of $M$ learnable ellipsoids. The network analytically computes the SDDF for a ray intersecting these ellipsoids in closed form.
   • Fusion: For a union of ellipsoids, the prior selects the closest intersected ellipsoid to determine the distance, ensuring the model handles occlusions naturally.
   • Initialization: A specialized algorithm (Algorithm 2) initializes ellipsoids by merging K-means clusters on planar surfaces (walls, floors) to avoid over-segmentation, leaving more ellipsoids for complex objects.

2. Implicit Neural Residual Network ($L$ and $R$):

   • Purpose: Refines the coarse prior to capture fine geometric details.
   • Latent Feature Network ($L$): Takes the intersection point $q$ (on the ellipsoid surface), the local viewing direction $v'$, and the ellipsoid index as input. It uses a polynomial embedding (degree-2 monomials of $q$ and $v'$) to generate a latent feature vector $z$.
   • Residual Decoder ($R$): A Multi-Layer Perceptron (MLP) that decodes $z$ into a correction term $\delta f$.
   • Final Prediction: $\hat{f}(p, v) = f_{prior}(p, v) + \delta f(p, v)$.

C. Training and Optimization

• Loss Function: Uses a weighted sum of Huber losses for intersection indicators, sign indicators, and distance values. Negative samples (points inside objects) are augmented to ensure the ellipsoids cover the scene.
• Differentiability: The entire pipeline, including the ellipsoid intersection logic and the residual network, is differentiable. This enables gradient-based viewpoint optimization (e.g., Next-Best-View planning) without iterative sphere tracing.

3. Key Contributions

1. Scene-Level SDDF Definition: Extends the SDDF concept from single objects to complex, large-scale scenes, handling occlusions and sign changes effectively.
2. Hybrid Explicit-Implicit Model: Proposes a novel architecture combining analytically derived ellipsoid priors (for coarse structure and occlusion handling) with neural residuals (for fine details).
3. Theoretical Guarantees: Proves that the combined model satisfies the Directional Eikonal equation by construction, eliminating the need for additional regularization losses.
4. Efficient Initialization: Develops an algorithm to initialize ellipsoids that efficiently approximates planar surfaces, reducing the number of parameters needed.
5. Differentiable View Optimization: Demonstrates the ability to optimize camera trajectories for maximum coverage and minimum overlap using the differentiable SDDF.

4. Experimental Results

The method was evaluated on Replica and Gibson datasets (synthetic LiDAR and RGB-D) and ScanNet (real RGB-D).

• Accuracy:
  • Achieves competitive Mean Absolute Error (MAE) in SDDF prediction, comparable to or better than state-of-the-art SDF and NeRF methods.
  • Outperforms RaDe-GS and Nerfacto in geometric consistency, particularly in areas with sparse observations or complex boundaries.
  • Outperforms SDF-Instant-NGP in preserving sharp details (e.g., plants, chair edges) which sphere tracing tends to smooth out.
• Efficiency:
  • Inference Speed: Significantly faster than SDF-based sphere tracing (1.5x faster) and NeRF (4x faster) because it requires only a single forward pass.
  • Memory: Uses significantly less GPU memory (3.3GB training vs. 18.7GB for RaDe-GS) due to fewer parameters (2.7M vs. 113.7M).
• Ablation Studies:
  • Negative Samples: Crucial for training; without them, ellipsoids fail to cover objects, leading to large errors.
  • Polynomial vs. Fourier Embedding: The proposed polynomial embedding generalizes better across scenes than Fourier features, which require scene-specific hyperparameter tuning.
  • Initialization: The proposed multi-ellipsoid initialization outperforms standard K-means++ by better handling planar surfaces.

5. Significance

This work bridges the gap between the geometric precision of SDFs and the efficiency of neural radiance fields.

• Geometric Fidelity: It provides a robust method for learning accurate 3D geometry directly from sensor data without relying on photometric supervision alone.
• Real-Time Applications: The single-query nature of SDDF makes it highly suitable for real-time robotics applications, such as active perception, navigation, and trajectory optimization, where differentiable distance queries are required.
• Scalability: The hybrid approach allows for scaling to large scenes with lower computational costs compared to dense neural fields or iterative tracing methods.

In summary, the paper presents a mathematically grounded, efficient, and highly accurate framework for 3D scene reconstruction that enables differentiable geometric reasoning at the scene level.