Technical Summary: Mind the Gap – Geometrically Accurate Generative Reconstruction from Disjoint Views

1. Problem Definition

The paper addresses a fundamental limitation in current 3D computer vision: the strict reliance on visual overlap between input images. Traditional reconstruction methods (e.g., Structure-from-Motion, Neural Radiance Fields, 3D Gaussian Splatting) and modern feed-forward estimators (e.g., Dust3R, AnySplat) require shared visual features to establish geometric correspondences and align coordinate systems. Conversely, purely generative 3D models (e.g., VistaDream, LucidDreamer) can hallucinate unobserved spaces but often fail to enforce strict geometric consistency across multiple independent, real-world observations.

This creates a critical bottleneck in real-world scenarios such as swarm robotics, consumer-level virtual tours, and crowdsourced data collection, where cameras often capture disjoint views (zero-overlap) to maximize coverage. In these "zero-overlap" scenarios, existing state-of-the-art methods fail catastrophically, producing disconnected geometric clusters, unaligned coordinate frames, or semantically incoherent reconstructions.

The authors formalize a new task: Generative Reconstruction from Disjoint Views. The goal is to synthesize a single, unified 3D environment from entirely non-overlapping images, treating original observations as hard geometric constraints while generating plausible intermediate spaces.

2. Methodology: The GLADOS Framework

To solve this, the authors propose GLADOS (Generative Reconstruction from Disjoint Views), a modular, three-stage framework that integrates foundation model reasoning with geometric rigor. The pipeline operates as follows:

Stage 1: Generative Spatial Bridging

Since disjoint views ($I_1, I_2$) share no visual overlap ($\Omega(I_1, I_2) = \emptyset$), the system first synthesizes an intermediate "anchor" image ($I_b$) to create a continuous visual link.

• Reasoning: A Vision-Language Model (VLM) analyzes the structural semantics of $I_1$ and $I_2$ to generate a descriptive text prompt ($p$) encapsulating the probable intermediate geometry.
• Synthesis: A multi-conditional diffusion model ($P_{diff}$) uses $I_1, I_2,$ and $p$ to generate a pool of candidate intermediate images.
• Selection: A secondary VLM evaluator ($P_{eval}$) acts as a zero-shot reward model to select the optimal anchor image ($I_b$) based on spatial and semantic coherence.
• Result: The input set becomes a triplet $\{I_1, I_b, I_2\}$, where adjacent pairs now share non-empty visual intersections, enabling geometric registration.

Stage 2: Robust Coarse 3D Reconstruction

The synthesized anchor $I_b$ may contain generative noise or hallucinations. The system must establish a coarse geometric scaffold ($S_c$) robust to these inconsistencies.

• Approach: The framework employs a dense pointmap regression model ($P_{geo}$, e.g., Dust3R) to perform global alignment directly in 3D coordinate space.
• Mechanism: Instead of minimizing noise-sensitive 2D reprojection errors, $P_{geo}$ treats reconstruction as a direct regression of dense 3D pointmaps. This acts as a spatial low-pass filter, absorbing local generative contradictions.
• Output: The aligned point cloud is parameterized as an initial set of 3D Gaussians, forming a coarse scaffold.

Stage 3: Iterative Context Expansion and Consistency Optimization

The coarse scaffold is incomplete and may contain "floaters" or holes. This stage fills unobserved regions while enforcing consistency.

• Expansion: The system renders the scene from novel viewpoints along an interpolated trajectory. For views with significant holes, an RGB inpainting prior ($P_{inp}$), conditioned on the text prompt $p$, synthesizes missing visual context.
• Depth Alignment: A monocular metric depth estimator ($P_{dpt}$) aligns the inpainted images with existing depth constraints before integrating new Gaussians.
• Consistency: To prevent drift and diverging geometries, a Multiview Consistency Sampling (MCS) approach using a Latent Consistency Model ($P_{lcm}$) is applied. This enforces multi-view constraints during the reverse diffusion process.
• Refinement (Anchored Grid Inpainting): To eliminate residual geometric voids and ensure metric accuracy, the framework employs an iterative refinement loop:
  1. Hole Detection: Renders opacity maps ($\alpha$) to identify unobserved regions.
  2. Anchored Inpainting: Constructs a composite grid containing novel views (with holes) and the original ground-truth images ($I_1, I_2$). A diffusion prior ($P_{mgi}$) inpaints the holes while strictly preserving the ground-truth anchors, enforcing semantic and color consistency.
  3. Super-Resolution & 3D Lift: A super-resolution prior ($P_{up}$) enhances fidelity. The enhanced 2D patches are lifted into 3D via metric depth estimation and affine alignment, injecting new 3D Gaussian primitives exclusively into the masked regions.
  4. Optimization: The updated field undergoes brief optimization to integrate new primitives.

3. Key Contributions

1. New Task Definition: The paper formalizes "Generative Reconstruction from Disjoint Views," a paradigm requiring systems to satisfy hard geometric constraints for observed regions while generating unobserved intermediate spaces.
2. Specialized Benchmark: The authors introduce a new benchmark dataset comprising 22 scenes (indoor/outdoor, diverse trajectories) sampled from existing datasets (Mip-NeRF 360, Tanks & Temples, etc.) and augmented with custom collections. Crucially, these inputs are strictly non-overlapping.
3. Evaluation Metrics: They propose reference-free metrics tailored for sparse ground truth, including GeCo (Geometric Consistency Metric) for detecting deformation/occlusion artifacts and MEt3R for measuring multi-view consistency robust to lighting changes, alongside standard FID and CLIP scores.
4. GLADOS Framework: A pioneering three-stage architecture that successfully unifies generative bridging, robust coarse initialization, and iterative consistency optimization.

4. Experimental Results

The authors benchmark GLADOS against state-of-the-art reconstruction methods (AnySplat, VGGT, Dust3R) and generative baselines (VistaDream) on the disjoint-view benchmark.

• Baseline Performance: Existing methods fail catastrophically on zero-overlap inputs. Purely geometric methods (Dust3R, VGGT) produce disconnected geometries or fail to register frames. Purely generative methods (VistaDream) produce lower FID scores but often result in structurally inconsistent or topologically broken scenes.
• GLADOS Performance: GLADOS achieves the best overall performance across geometric metrics (MEt3R, GeCoM, GeCoD, GeCoF) and perceptual metrics (CLIP, FID). It is the only method that achieves zero reconstruction failures across all evaluated scenes.
• Ablation Studies:
  • Removing Generative Spatial Bridging (w/o Midpoint) leads to 4 reconstruction failures, as dense regression cannot find correspondences.
  • Removing Iterative Context Expansion (w/o ICE&CO) preserves input fidelity (higher CLIP) but degrades structural integrity (higher FID, GeCo errors).
  • Removing Refinement (w/o Refinement) leaves residual geometric voids and misalignment, resulting in higher photometric and consistency errors.

5. Significance and Claims

The paper claims that GLADOS represents a significant step forward by bridging the gap between geometric reconstruction (which ensures metric accuracy) and generative synthesis (which enables completion of missing data).

• Paradigm Shift: It moves beyond the assumption that 3D reconstruction requires overlapping views, enabling applications in distributed robotics and crowdsourced data where overlap is impossible.
• Robustness: By treating original disjoint observations as "hard constraints" and using generative models only for the intermediate space, GLADOS avoids the geometric distortions typical of purely generative approaches.
• Modularity: The framework is architecture-agnostic, allowing for the seamless integration of future advances in foundation models, reconstruction algorithms, and inpainting techniques.

The authors modestly note a limitation: the framework relies on generative priors, which may introduce semantically plausible but geometrically inaccurate structures in heavily unconstrained regions where observational constraints are insufficient. However, the proposed anchored grid inpainting and metric grounding aim to mitigate this.