Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself

1. Problem Statement

Feed-forward multi-view 3D reconstruction models (e.g., Depth Anything 3, VGGT) have achieved impressive zero-shot performance, enabling real-time 3D scene reconstruction without explicit pose estimation. However, these models suffer from a "train-then-freeze" rigidity:

• Lack of Adaptability: Once trained, their parameters are fixed. They cannot adapt to novel test scenes, leading to geometric errors in challenging conditions like occlusions, specularities, and ambiguous textures.
• Data Scarcity: Collecting large-scale, high-quality 3D ground truth for diverse real-world environments to retrain or fine-tune these models is prohibitively expensive and often impossible.
• The Gap: Existing Test-Time Adaptation (TTA) methods often rely on weak self-supervised signals (e.g., entropy minimization) or symmetric consistency losses that may regress to the mean, failing to provide a strong supervisory signal for geometric refinement.

2. Core Insight & Methodology

The authors propose Free Geometry, a plug-and-play TTA framework that enables feed-forward models to "self-evolve" at test time without 3D ground truth.

Key Insight: "More Views = Better Geometry"

The authors observe that in multi-view reconstruction, providing more input views strengthens geometric constraints and reduces ambiguity. Consequently, a model processing a full sequence (e.g., 8 views) produces more reliable and view-consistent feature representations than when processing a masked subset (e.g., 4 views). This creates a natural, label-free hierarchy where the full-view prediction acts as a "teacher" for the partial-view "student."

Technical Framework

Free Geometry operates by enforcing consistency between a frozen "Teacher" branch and a trainable "Student" branch:

1. Architecture Setup:

   • Teacher Branch: Processes the full observation (all $N$ views) using the frozen backbone. It generates high-quality feature representations ($F_{full}$).
   • Student Branch: Processes a partial observation (masked subset, e.g., $M < N$ views). It uses the same backbone but is augmented with LoRA (Low-Rank Adaptation) modules and trainable camera tokens. It generates student features ($F_{partial}$).
   • Optimization Target: Only the LoRA parameters and camera tokens are updated; the backbone remains frozen.

2. Self-Supervised Geometric Losses:
   The method optimizes the student to match the teacher's features using two complementary losses:

   • Intra-frame Consistency Loss: Aligns the student's features with the teacher's features at corresponding spatial locations (patch tokens and camera tokens) on the unmasked frames. It combines a Huber loss (magnitude) and cosine similarity (direction).
   • Cross-frame Relational Loss: Preserves the geometric relationships implied by the masked frames. It constructs triplets of tokens (two unmasked, one masked anchor) and enforces that the student preserves the pairwise similarity distributions and the virtual triangle angles (cosine angles) found in the teacher's feature space. This transfers structural knowledge from the unobserved views.

3. Efficiency:
   The adaptation is extremely fast, taking less than 2 minutes per dataset on a single GPU, as it only updates a tiny fraction of parameters (<0.2%) via LoRA.

3. Key Contributions

1. Identification of a "More-Views-Better" Regime: The paper establishes that full-view predictions in foundation models serve as a superior, architecture-guaranteed supervisor for partial-view predictions, providing a free self-supervision signal.
2. Free Geometry Framework: A novel TTA method that performs feature-level self-distillation. Unlike previous methods that align outputs, Free Geometry aligns encoder features, addressing the representation bottleneck directly.
3. Dual-Level Geometric Recalibration: The introduction of a dual-loss objective (intra-frame consistency + cross-frame relational constraints) that effectively transfers geometric structure from masked views to the student model.
4. Generalization: The method is model-agnostic (plug-and-play) and generalizes across different input view counts (e.g., adapting on 8→4 views improves performance on 4, 8, 16, and 32 view inputs).

4. Experimental Results

The method was evaluated on four diverse benchmarks: ETH3D, ScanNet++, 7-Scenes, and HiRoom, using Depth Anything 3 and VGGT as baselines.

• Quantitative Improvements:
  • Camera Pose Accuracy: Average improvement of 3.73% in AUC@3.
  • Point Map Prediction: Average improvement of 2.88% in F1-score.
  • Low-View Regime: Gains are most significant when input views are scarce (e.g., 4 views), where geometric constraints are weakest.
• Cross-View Generalization: Models adapted using an 8→4 view consistency signal showed consistent improvements even when tested with 16 or 32 views, proving the adaptation improves the model's internal geometric reasoning rather than overfitting to a specific view count.
• Qualitative Results: Visualizations show reduced depth errors, fewer scattered outliers, and cleaner surface structures, particularly in regions with occlusions and reflective surfaces.
• Ablation Studies:
  • Removing either the consistency loss or the relational loss significantly degrades performance, confirming the necessity of both terms.
  • The "mixed selection" strategy for masked anchors (using both most and least similar patches) outperforms random or purely top-k selection.

5. Significance

Free Geometry addresses a critical limitation in modern 3D vision: the inability of powerful foundation models to adapt to the specific nuances of a new scene without ground truth.

• Practicality: It offers a low-cost, high-efficiency solution for deploying 3D reconstruction models in real-world scenarios where data is unlabeled and scenes are novel.
• Paradigm Shift: It moves beyond the rigid "train-then-freeze" paradigm by enabling test-time self-evolution using the model's own internal consistency as a supervisory signal.
• Scalability: By leveraging LoRA, it allows massive foundation models to be adapted on consumer-grade hardware with minimal memory overhead, making robust 3D reconstruction accessible for dynamic, unlabeled environments.