SceneExpander: Expanding 3D Scenes with Free-Form Inserted Views

1. Problem Definition

The paper addresses the challenge of 3D scene expansion in a user-centric workflow. Unlike traditional 3D reconstruction which assumes all input views are geometrically consistent, this work tackles a scenario where:

• Input: A set of captured multi-view images of a real scene ($C$) and a single inserted novel view ($I_g$) generated by an AI image model (e.g., via text prompts).
• The Conflict: The inserted view $I_g$ is often 3D-misaligned with the original captured geometry. It may contain hallucinated content, geometric shifts, or inconsistencies that violate the multi-view constraints of the original scene.
• The Goal: Reconstruct an expanded 3D scene that:
  1. Preserves the fidelity of the original captured region (preventing "catastrophic drift").
  2. Integrates the new content from $I_g$ to extend the scene beyond the observed boundary.
  3. Maintains view consistency for novel viewpoints within the expanded region.

Existing methods fail here because they either suppress the new content to maintain consistency or degrade the original reconstruction when forced to fit the misaligned view.

2. Methodology: SceneExpander

The authors propose SceneExpander, a framework based on Test-Time Adaptation (TTA). Instead of retraining the model or requiring 3D-consistent generation, it adapts a pre-trained feed-forward reconstruction model (specifically WorldMirror) at inference time using two complementary distillation signals.

Core Architecture

• Backbone: A parametric feed-forward model $f_\theta$ that predicts camera parameters, depth, normals, and a 3D Gaussian Splatting (3DGS) representation from input views.
• Teacher-Student Setup:
  • Student ($\theta$): The model being adapted at test time.
  • Anchor Teacher ($\theta_A$): Frozen initialization of the student. Provides a "conservative prior" for the captured views.
  • Generation Teacher ($\theta_G$): An Exponential Moving Average (EMA) of the student, updated iteratively to provide soft labels for the inserted view.

Key Components

1. Anchor Distillation (Stabilization):

   • Purpose: Prevent the model from distorting the original scene while trying to fit the new view.
   • Mechanism: The student is forced to match the geometric predictions (camera pose, depth, normals) of the frozen Anchor Teacher on the captured views.
   • Loss: $\mathcal{L}_{anchor}$ minimizes the distance between student and teacher predictions on the original set $C$.

2. Inserted-View Self-Distillation (Adaptation):

   • Purpose: Integrate the misaligned $I_g$ without treating its pixels as hard ground truth (which would cause artifacts).
   • Mechanism: The Generation Teacher ($\theta_G$) provides a "soft" geometric target for $I_g$. As the student improves, the EMA teacher updates, gradually refining the target.
   • Loss: $\mathcal{L}_{gen}$ distills the student's predictions on $I_g$ toward the teacher's soft labels.
   • Strategy: The inserted view is included in the training batch with a probability $p$ (default 0.5) to prevent overfitting to the single noisy view.

3. Geometry-Perturbation Augmentation:

   • To improve robustness against spatial inconsistencies, the method applies subset sampling (randomly dropping captured views) and geometry-perturbation augmentation (global affine warps and blockwise piecewise-affine warps) to the input batch. This simulates the types of misalignments found in generative outputs.

4. Stochastic Restoration:

   • To prevent long-horizon drift during continuous adaptation, a random subset of parameters is periodically restored toward the initial pre-insertion weights ($\theta_0$).

Optimization Objective

The total loss function balances preservation and expansion:
$$\min_{\theta} \lambda_A \mathcal{L}_{anchor} + \lambda_G \mathcal{L}_{gen} + \lambda_{reg} \mathcal{L}_{reg}$$
Where $\lambda_A$ is high (to protect the original scene) and $\lambda_G$ is low (to treat the new view as a soft constraint).

3. Key Contributions

1. Task Formulation: Defined the novel task of 3D scene expansion via free-form view insertion, bridging text-driven generation, view insertion, and editable 3D world building.
2. SceneExpander Framework: Introduced a test-time tuning approach that uses Anchor Distillation and Self-Distillation to robustly integrate 3D-misaligned views without retraining.
3. Robustness Mechanisms: Developed specific techniques (subset sampling, geometry perturbation, stochastic restoration) to handle the inherent noise and inconsistency of AI-generated views.

4. Experimental Results

The method was evaluated on the ETH dataset (outdoor) and diverse online collections (indoor/outdoor).

• Baselines: Compared against WorldMirror, OmniVGGT, TTT3R, and standard 3DGS optimization.
• Metrics:
  • Preservation: PSNR, SSIM, LPIPS on original captured views.
  • Expansion Quality: GPT-based scoring (Insertion Satisfaction, Rendering Quality) and Human Evaluation (H-IS, H-RQ).
• Performance:
  • Preservation: SceneExpander achieved the highest PSNR (21.21) and SSIM (0.776), significantly outperforming baselines (e.g., WorldMirror: 19.30 PSNR). This confirms it successfully protects the original scene.
  • Expansion: It achieved the best human scores (H-IS: 4.56, H-RQ: 4.53) and strong GPT scores, indicating the inserted content is visually plausible and navigable.
  • Qualitative: Baselines like 3DGS and TTT3R suffered from severe ghosting and artifacts when forced to fit misaligned views. SceneExpander produced coherent geometry and appearance in the expanded regions.

5. Significance and Limitations

• Significance: This work shifts the paradigm from "perfect 3D-consistent generation" (which is expensive and difficult) to "robust integration." It enables iterative, interactive world building where users can expand scenes using simple text prompts and 2D generative models, even if the generated views are not geometrically perfect.
• Limitations: The method relies on test-time adaptation to mitigate conflicts. If the initial inconsistency between the inserted view and the scene is too severe (e.g., completely different structures), the model may still produce artifacts, as seen in the failure case involving a shrine plaque.

In summary, SceneExpander provides a practical solution for expanding 3D worlds by treating generative views as "soft constraints" rather than hard truths, allowing for flexible, user-controlled scene extension while maintaining the integrity of the original capture.