GeoFusionLRM: Geometry-Aware Self-Correction for Consistent 3D Reconstruction

1. Problem Statement

Single-image 3D reconstruction using Large Reconstruction Models (LRMs) has advanced significantly, yet current state-of-the-art methods suffer from geometric inconsistencies and misaligned details.

• The Core Issue: While LRMs (e.g., InstantMesh, LRM) can generate coarse 3D shapes, they often produce meshes with inaccurate geometry, distorted surface normals, and details that do not align with the conditioning input image.
• Limitations of Existing Approaches:
  • Semantic Reliance: Most models rely solely on semantic features extracted from the input image via attention mechanisms, lacking explicit geometric feedback.
  • View Synthesis Artifacts: Methods that synthesize multi-view images to create stronger priors often introduce geometric inconsistencies between views.
  • Optimization Overhead: Optimization-based methods (e.g., SDS) achieve high fidelity but are computationally expensive and slow.
  • External Dependencies: Some approaches require additional external networks for depth/normal estimation, increasing complexity.

The paper addresses the need for a self-contained framework that improves geometric fidelity and consistency without external supervision or heavy optimization.

2. Methodology: GeoFusionLRM

The authors propose GeoFusionLRM, a geometry-aware self-correction framework that integrates a two-stage refinement process into a baseline LRM (specifically InstantMesh). The core idea is to use the model's own intermediate predictions (depth and normals) to correct its own reconstruction errors.

Architecture Overview

The pipeline consists of two main passes:

1. Initial Reconstruction: The baseline LRM generates an initial mesh ($M^{(0)}$) from the input image.
2. Self-Correction Pass:
   • Geometric Cues Extraction: The initial mesh is rendered to produce depth ($D$) and normal ($N$) maps.
   • GeoFormer Encoder: A dedicated geometry-aware encoder processes these $D$ and $N$ maps. It is initialized from a pretrained ViT (DINO) and fine-tuned with geometric supervision to capture structural consistency.
   • GeoFuser Module: A lightweight token-wise fusion network merges the semantic tokens (from the original image) with the geometry-aware tokens (from GeoFormer).
   • Refinement: The fused tokens are injected back into the LRM's cross-attention layers to generate an updated, more accurate mesh.

Key Components

• GeoFormer Encoder:
  • Initialized as a copy of the ViT encoder used in InstantMesh.
  • Modified to accept 4-channel inputs (Depth + Normal) instead of 3-channel RGB, with zero-initialized weights for the new channel.
  • Trained to encode structural information from geometric maps.
• GeoFuser Module:
  • A two-layer feed-forward network with SiLU activation.
  • Computes a residual correction: $F_{fused} = F_{sem} + f_\theta(F_{sem}, F_{geo})$.
  • Zero-initialization strategy: The final linear layer is initialized with zero weights, ensuring the correction is disabled in the first iteration (acting as the baseline) and enabled only during refinement.

Training Strategy

• Freezing: The baseline InstantMesh backbone is frozen; only GeoFormer and GeoFuser parameters are optimized.
• Loss Function: Combines photometric MSE, perceptual LPIPS, mask consistency, depth alignment, normal similarity, and FlexiCubes regularization.
• Unrolling: During training, the refinement process is unrolled for $T=3$ steps. At inference, a single refinement step (2 total passes) is found to be sufficient.

3. Key Contributions

1. Self-Predicted Geometry-Aware Conditioning: Introduced a framework that refines meshes by conditioning on depth and normal cues predicted from intermediate reconstructions, eliminating the need for external predictors.
2. GeoFormer Encoder: A specialized encoder fine-tuned with geometric supervision to capture structural alignment, initialized from DINO for robust feature extraction.
3. GeoFuser Module: A lightweight, token-level fusion network that effectively merges semantic and geometric features to produce refined triplane conditioning.
4. Improved Consistency and Fidelity: Demonstrated that this two-stage process yields sharper geometry, more accurate normals, and better alignment with input views compared to existing baselines.

4. Experimental Results

The method was evaluated on Objaverse-1.0 (training), OmniObject3D, and Google Scanned Objects (GSO) (testing).

Quantitative Performance

• Metrics: PSNR, SSIM, and LPIPS for both RGB images and Normal maps.
• Results: GeoFusionLRM consistently outperformed baselines (OpenLRM, SPAR3D, LGM, InstantMesh) across all datasets and metrics.
  • Normal Maps: The most significant improvements were observed in normal map metrics (e.g., +0.003 SSIM and -0.0033 LPIPS over InstantMesh on GSO), indicating superior geometric accuracy.
  • RGB: Modest but consistent improvements in visual fidelity.

Qualitative Performance

• Detail Preservation: The method successfully preserved fine details (e.g., sharp edges of bowls, cut-out patterns on planters) that other models smoothed over or distorted.
• Handling Ambiguity: In low-feature or dark regions (e.g., a hat's dark band), GeoFusionLRM reconstructed coherent geometry, whereas others hallucinated holes or flattened the shape.
• Limitations: The method still struggles with extremely fine structures (e.g., thin root segments) due to the resolution limits of the underlying triplane representation, though it improves upon the baseline's handling of coarse branches.

Computational Cost

• Trade-off: The refinement process increases computational cost.
  • TFLOPs: Increased from ~3.88 (InstantMesh) to ~8.69.
  • Inference Time: Increased from ~0.99s to ~3.85s per object.
• Efficiency: Despite the cost, the authors argue that a single refinement step offers the optimal trade-off between quality and efficiency.

5. Significance

GeoFusionLRM represents a significant step forward in self-corrective 3D reconstruction.

• Paradigm Shift: It moves away from relying solely on semantic priors or external view synthesis, introducing an internal feedback loop where geometry informs the reconstruction process.
• Practical Impact: By improving normal accuracy and geometric consistency, it enables more reliable 3D assets for applications in AR/VR, robotics, and content creation where physical plausibility is critical.
• Efficiency vs. Quality: It offers a middle ground between fast but inaccurate feed-forward models and slow, high-fidelity optimization-based methods, providing a practical solution for high-quality single-image reconstruction.