Neu-PiG: Neural Preconditioned Grids for Fast Dynamic Surface Reconstruction on Long Sequences

1. Problem Statement

Reconstructing temporally consistent surfaces from unstructured dynamic point clouds (e.g., humans, animals) remains a significant challenge, particularly for long sequences. Existing approaches face a trade-off:

• Optimization-based methods (e.g., DynoSurf, PDG) produce high-quality results but suffer from long runtimes and risk "drift" (accumulated errors) over time.
• Learning-based methods offer fast inference but rely on category-specific priors (e.g., SMPL for humans), limiting their ability to generalize to arbitrary shapes or unseen categories.
• Current limitations: There is a lack of a general, efficient, and category-agnostic method that achieves high-fidelity, drift-free reconstruction on long sequences without requiring explicit correspondences or heavy pre-training.

2. Methodology: Neu-PiG

Neu-PiG (Neural Preconditioned Grids) proposes a fast optimization-based framework that learns spatially smooth and temporally coherent deformations. The core pipeline involves:

A. Initialization and Reference Mesh

• A keyframe ($t_{key}$) is selected from the input sequence.
• A reference surface mesh ($X_{t_{key}}$) is reconstructed from the point cloud at this frame using screened Poisson surface reconstruction. This mesh serves as the canonical topology for all subsequent frames.
• The method assumes the topology remains fixed; it learns the deformation of this mesh over time.

B. Multi-Resolution Preconditioned Latent Grids

Instead of optimizing deformation fields directly or using dense neural networks for every point, Neu-PiG encodes deformations into a multi-resolution latent grid:

• Dual Grids: Two distinct voxel grids are used:
  1. Position Grid ($G_p$): Encodes spatial context based on vertex positions. It is a multi-resolution hierarchy (8 levels, from $2^3$ to $32^3$ cells) storing 30-dimensional feature vectors.
  2. Normal-Direction Grid ($G_n$): Encodes local orientation information based on vertex normals. It is a single-resolution grid ($4^3$) storing 2-dimensional feature vectors.
• Aggregation: For any query point, features are retrieved via trilinear interpolation and aggregated across all levels of the position grid to form a unified latent representation. This ensures both global consistency and local high-frequency detail.

C. Temporal Deformation Model

• Input: For each vertex, the model combines the latent features ($z_p, z_n$) with a Fourier time embedding ($\gamma(t)$) that encodes the timestep.
• Network: A lightweight Multi-Layer Perceptron (MLP) $\psi$ takes this combined input and predicts a 6-DoF deformation per vertex:
  • A rotation quaternion ($q \in \mathbb{R}^4$).
  • A displacement vector ($d \in \mathbb{R}^3$).
• Transformation: The predicted rotation and displacement are applied to the reference mesh vertices to reconstruct the surface for the current frame.

D. Sobolev Preconditioning (Key Innovation)

To ensure stability and convergence during gradient-based optimization, Neu-PiG applies Sobolev preconditioning to the latent grid updates:

• Instead of updating raw gradients, the method applies a low-pass filter (based on the Laplacian matrix of the grid) to the gradients.
• This enforces spatial smoothness on the latent features, preventing high-frequency noise and ensuring that deformations remain coherent across the sequence.
• This allows the method to avoid explicit correspondences or strong priors while preventing drift.

E. Optimization Objectives

The system jointly optimizes the grid features and MLP weights using:

1. Deformation Loss ($L_{def}$): Robust Chamfer Distance between the reconstructed mesh and the input point cloud, weighted by a time-adaptive confidence term.
2. Isometry Loss ($L_{iso}$): Penalizes changes in edge lengths to preserve local surface structure.

3. Key Contributions

1. Fast Optimization Framework: A method that estimates temporally coherent deformations for arbitrary subjects (humans, animals) from sequential point clouds, running 60× faster than prior training-free optimization methods.
2. Novel Preconditioned Encoding: A surface encoding parameterized by vertex positions and normal directions, storing deformations in a unified latent space without relying on correspondences.
3. Multi-Scale Latent Grid: A hierarchical representation that enforces spatial consistency at both global and local scales, enabling fast decoding via a lightweight MLP.
4. Sobolev Preconditioning: The application of Sobolev gradient methods to latent voxel spaces, ensuring smooth, drift-free updates over long sequences.

4. Experimental Results

The authors evaluated Neu-PiG on three benchmarks: DFAUST (human motion), AMA (clothed humans), and DT4D (articulated animals).

• Accuracy: Neu-PiG achieves state-of-the-art (SOTA) performance in Chamfer Distance (CD), Normal Consistency (NC), and F-score across all datasets. For example, on DFAUST, it achieves a CD of 0.39 (vs. 0.46 for PDG and 2.13 for DynoSurf).
• Speed:
  • Training-free methods: Neu-PiG is 60× faster than PDG and DynoSurf (e.g., 32 seconds vs. 7 minutes for a sequence).
  • Inference: It achieves inference speeds comparable to heavy pre-trained models.
• Scalability: On long sequences (up to 120 frames), Neu-PiG maintains stable correspondences and high fidelity, whereas competitors like PDG and DynoSurf show significant degradation or failure due to drift.
• Ablation Studies:
  • Removing preconditioning or multi-resolution grids significantly degrades performance.
  • Fourier time encoding outperforms polynomial or learned embeddings.
  • The method is robust to input noise (up to 0.5%) and benefits from higher point cloud densities.

5. Significance

Neu-PiG bridges the gap between the accuracy of optimization-based methods and the efficiency of learning-based approaches. By introducing Neural Preconditioned Grids, it eliminates the need for category-specific priors (making it applicable to any deformable object) while solving the drift problem inherent in long-sequence reconstruction.

Its ability to produce high-fidelity, temporally consistent 4D reconstructions in seconds makes it a scalable foundation for real-time applications in Augmented/Virtual Reality (AR/VR), robotics, autonomous systems, and dynamic scene understanding, where handling long, unstructured sequences of arbitrary objects is critical.