Learning 3D Reconstruction with Priors in Test Time

1. Problem Statement

The Challenge:
Current state-of-the-art feed-forward 3D reconstruction methods, specifically Multiview Transformers (MVTs) (e.g., VGGT, $\pi^3$), achieve impressive results by processing multiple RGB images in a single forward pass. However, they are natively designed to accept only RGB images as input.

• Limitation: In real-world scenarios, valuable auxiliary information (priors) is often available, such as camera poses (from SLAM or rig systems), intrinsics, or depth maps (from LiDAR/stereo).
• Current Solutions' Flaw: Existing methods that incorporate these priors (e.g., Pow3R, MapAnything) require modifying the network architecture to accept priors as additional input channels. This necessitates retraining the entire model whenever the backbone changes or the type of available prior changes, making these solutions inflexible and computationally expensive.

The Goal:
Develop a framework that can leverage available priors (camera poses, intrinsics, depth) to improve 3D reconstruction tasks without retraining the pre-trained MVT or altering its architecture.

2. Methodology: Test-time Constrained Optimization (TCO)

The authors propose Test-time Constrained Optimization (TCO), a plug-and-play framework that treats priors not as inputs to the network, but as constraints on the network's outputs.

Core Concept

Instead of feeding priors into the MVT, the method optimizes the network parameters at inference time to minimize a loss function that enforces consistency between the model's predictions and the available priors.

Key Components

A. Optimization Formulation
The problem is framed as minimizing a self-supervised loss subject to constraints:
$$\min_{\theta} \mathcal{L}(\text{MVT}_\theta(I)) + \sum_{m=1}^{M} \mu_m \|g_m(\text{MVT}_\theta(I))\|$$
Where:

• $\theta$: Network parameters (specifically the shared decoder).
• $\mathcal{L}$: A self-supervised objective ensuring multi-view consistency.
• $g_m$: Penalty terms representing the deviation of predictions from the priors (e.g., predicted pose vs. known pose).
• $\mu_m$: Hyperparameters controlling the weight of each constraint.

B. The Objective Function: Prediction Compatibility
To prevent the model from simply overfitting to the priors and degrading the overall 3D quality, a self-supervised objective is used. This measures the compatibility between predictions across different views.

• Mechanism: The method uses 2D Gaussian Splatting (2DGS) as a differentiable renderer.
• Process:
  1. Split views into source and target.
  2. Derive 2DGS parameters (position, color, normal, opacity, rotation) directly from the MVT's predicted depth, pose, and intrinsics.
  3. Render the source views into the target views.
  4. Calculate Photometric Loss (for depth/point map tasks) or Geometric Loss (for pose tasks) between the rendered images and the actual target images.
• Heuristics: The 2DGS parameters are inferred heuristically from MVT outputs (e.g., confidence maps determine opacity) rather than being optimized directly, ensuring the optimization focuses on the MVT parameters.

C. Prior Penalty Terms
Priors are converted into specific loss terms:

• Camera Pose: Decomposed into rotation (angular distance) and translation (normalized by estimated scene scale).
• Intrinsics: L1 distance between predicted and prior focal lengths.
• Depth: Global scale and shift alignment followed by L1 distance.

D. Fine-Tuning Strategy (Task Synergy)
To maximize the benefit of priors across different tasks (e.g., using known poses to improve depth):

• Freeze: All task-specific prediction heads (depth head, pose head, etc.) and the encoder.
• Fine-tune: Only the shared decoder network using LoRA (Low-Rank Adaptation).
• Rationale: This allows the shared scene representation to adapt to the constraints without overfitting to specific heads, leveraging the synergy between modalities.

3. Key Contributions

1. TCO Framework: A novel test-time optimization approach that incorporates scene/camera priors into pre-trained MVTs without architectural changes or retraining.
2. Loss Design: A dual-component loss function combining:
   • Self-supervised compatibility: Enforces multi-view consistency via 2DGS rendering.
   • Prior penalties: Enforces adherence to available ground-truth priors.
3. Plug-and-Play Efficiency: The method is model-agnostic (works with VGGT, $\pi^3$, etc.) and requires only inference-time computation (LoRA fine-tuning).
4. Superior Performance: Demonstrates that optimizing at test time with constraints outperforms methods that retrain models with priors as inputs.

4. Experimental Results

The method was evaluated on standard 3D vision benchmarks: ETH3D, DTU, 7-Scenes, NRGBD (Point Map Estimation) and ScanNet (Camera Pose Estimation).

Quantitative Results:

• Point Map Estimation:
  • On ETH3D, TCO reduced point-map distance error by >50% compared to base image-only models (e.g., VGGT).
  • TCO consistently outperformed retrained prior-aware baselines (Pow3R, MapAnything) across all datasets, even when those baselines had access to the same priors during training.
  • Ablation: The "compatibility objective" (self-supervised loss) was proven essential; without it, adding priors sometimes degraded performance (e.g., on DTU).
• Camera Pose Estimation:
  • On ScanNet, TCO improved Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) significantly compared to the base VGGT model, even with sparse views (6 input images) and wide baselines.
• Efficiency:
  • Performance scales with the number of test-time iterations (5 to 80), with diminishing returns after 40 iterations.
  • LoRA fine-tuning ensures the process is parameter-efficient.

Qualitative Results:

• Visual comparisons show TCO correcting structural errors (e.g., misaligned walls, fragmented objects) present in base image-only reconstructions.
• TCO successfully reconstructs coherent objects (e.g., a whole hand) where the base model produces fragmented parts.

5. Significance and Impact

• Bridging the Gap: TCO bridges the gap between the flexibility of classical optimization-based methods (which use priors) and the speed of modern feed-forward deep learning methods.
• Practical Deployment: It offers a practical solution for real-world applications (Robotics, AR/VR, Autonomous Driving) where sensor data (LiDAR, IMU) provides priors that can be instantly leveraged to boost 3D reconstruction quality without the cost of retraining large foundation models.
• Paradigm Shift: It challenges the prevailing notion that priors must be fed as inputs; instead, it demonstrates that treating priors as inference-time constraints is a more effective and flexible strategy for integrating domain knowledge into deep learning models.