💻 computer science

PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting

PanoPlane introduces a training-free method for high-fidelity sparse-view indoor 3D Gaussian Splatting that leverages $360^{\circ}$ panoramic completion guided by a novel Layout Anchored Attention Steering mechanism to replace unconstrained hallucination with grounded surface extrapolation, achieving state-of-the-art novel view synthesis without fine-tuning the underlying diffusion model.

Original authors: Adil Qureshi, Dongki Jung, Jaehoon Choi, Dinesh Manocha

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adil Qureshi, Dongki Jung, Jaehoon Choi, Dinesh Manocha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a room and take just a few photos from different corners. You want to create a perfect, 3D model of the entire room so you can look around from any angle, even the ones you never photographed.

The problem is that with only a few photos, the computer gets confused. It tries to guess what the rest of the room looks like, but often it "hallucinates" weird floating furniture, blurry walls, or geometry that doesn't make sense. It's like trying to finish a jigsaw puzzle when you only have three pieces; you might guess the picture is a beach, but the actual room is a kitchen.

PanoPlane is a new method that solves this by changing how the computer "thinks" about the missing parts. Here is how it works, using simple analogies:

1. The "360-Degree Panorama" Trick

Instead of trying to guess the missing pieces one photo at a time (like a narrow window), PanoPlane first creates a 360-degree panoramic image of the room. Think of this as taking a photo that wraps all the way around you, showing the floor, ceiling, and all four walls in a single, continuous circle.

However, because you only took a few photos, this panorama has huge black "holes" where the computer has no data.

2. The "Architect's Blueprint" (Layout Anchored Attention)

This is the paper's biggest innovation. Usually, when an AI tries to fill in a hole in a picture, it just guesses based on nearby colors. It might see a patch of wood and guess the hole is also wood, even if that hole is actually supposed to be a wall.

PanoPlane uses a special trick called Layout Anchored Attention Steering. Here is the analogy:

The Old Way: Imagine a painter filling in a blank spot on a canvas. They just look at the paint right next to the hole and copy it. If the hole is near a table, they might paint more table, even if the hole is actually on the ceiling.
The PanoPlane Way: Imagine the painter has a blueprint of the room's structure (walls, floors, ceilings). Before they paint a single stroke, they check the blueprint.
- If the hole is on the "wall" section of the blueprint, the painter is forced to look only at other parts of the wall to decide what color and texture to use.
- If the hole is on the "floor," they look only at the floor.

The paper calls this "anchoring." It forces the AI to ignore random guesses and instead "anchor" its attention to the specific geometric surface (wall, floor, or ceiling) that the hole belongs to. It ensures that if you are filling in a missing part of a wall, the AI extends the wall logically, rather than inventing a floating object.

3. The "Smart Assistant" (Vision-Language Model)

How does the computer know which part of the hole is a wall and which is a chair?

The system uses a "smart assistant" (a Vision-Language Model) to look at the few photos you took.
It identifies the flat surfaces: "That's a wall," "That's a floor," "That's a ceiling."
It ignores the clutter (like chairs or tables) for the purpose of building the room's structure.
It then maps these labels onto the panoramic image, telling the AI exactly which "blueprint" line to follow for every missing pixel.

4. The Result: A Perfect 3D Room

Once the AI fills in the panoramic image using these strict architectural rules, the result is a complete, 360-degree view of the room where the walls, floors, and ceilings connect perfectly.

The system then takes this completed panorama and turns it back into a 3D model (using a technique called 3D Gaussian Splatting). Because the panorama was built with such strict geometric rules, the final 3D model is incredibly accurate. You can now look at the room from angles you never photographed, and the walls will look solid and real, not blurry or floating.

Why is this special?

No Extra Training: The paper highlights that this method doesn't need to re-train the AI. It works by "steering" the AI's attention during the process, like giving a driver a GPS route instead of teaching them how to drive a new car.
Works with Very Few Photos: It can build a high-quality 3D room from as few as three photos.
Better than Previous Methods: Previous methods often created "ghosts" or weird textures in the missing areas. PanoPlane ensures that the missing parts are just logical extensions of the existing walls and floors.

In short, PanoPlane stops the AI from daydreaming about what the missing room might look like and forces it to act like a structural engineer, ensuring that every missing piece fits perfectly into the room's actual blueprint.

Technical Summary: PanoPlane

Problem Statement

Reconstructing indoor scenes from sparse viewpoints for novel view synthesis is critical for tasks like scene understanding and embodied navigation. While 3D Gaussian Splatting (3DGS) achieves photorealistic rendering with dense multi-view coverage, it struggles significantly with sparse inputs (e.g., 3–9 views). In these scenarios, large portions of the scene remain unobserved, leading to collapsed geometry, floating artifacts, and incoherent surfaces.

Existing sparse-view methods attempt to address this by leveraging diffusion priors to generate additional training views. However, these approaches face two primary limitations:

Limited Field of View: Perspective-based generation captures only small portions of the scene, often introducing geometric inconsistencies across views.
Lack of Structural Guidance: Current methods cannot directly steer the internal attention of diffusion models toward structurally relevant observed regions during generation. Consequently, unobserved regions often suffer from unconstrained hallucinations rather than grounded surface extrapolation.

Methodology: PanoPlane

PanoPlane proposes a geometry-aware approach that reconstructs closed room geometry via panoramic scene completion. Instead of generating fragmented perspective views, it leverages 360° panoramic completion conditioned on the full spatial layout of the scene.

Core Pipeline

Plane-Aware Initialization:
- An initial coarse 2D Gaussian Splatting (2DGS) model is trained on sparse inputs.
- A Vision-Language Model (VLM) classifies detected planar surfaces into layout planes (walls, floors, ceilings) and non-layout planes (furniture, objects).
- Depth maps are refined using ray-plane intersection for pixels assigned to layout surfaces, ensuring geometric consistency.
Panoramic Rendering and Selection:
- From the coarse 2DGS, equirectangular (ERP) panoramas are rendered at selected viewpoints.
- Unobserved regions appear as "holes" in the panorama.
- A scoring function selects the optimal panorama for completion based on the number of unobserved voxels visible, the hole-to-observed ratio, and the diversity of layout planes in the observed region.
Layout Anchored Attention Steering (Key Innovation):
- Hole Assignment: Unobserved tokens (holes) in the latent panoramic space are assigned to specific layout planes using two complementary strategies:
  - Geometry-based: Ray-plane intersection in 3D space.
  - Boundary-based: 2D proximity to observed tokens on the latent grid.
- Attention Modification: A training-free mechanism modifies the Query ( $Q$ $Q$ ) and Key ( $K$ $K$ ) matrices within the diffusion transformer (DiT360) during inference.
  - For each hole token, the query vector is shifted toward the centroid of the key vectors of observed tokens belonging to the same assigned layout plane.
  - The keys of the observed tokens on that plane are reinforced.
- This forces the model to attend to geometrically consistent observed content rather than relying on statistical proximity, effectively replacing hallucination with grounded surface extrapolation.
Refinement:
- The completed 360° panoramas are decomposed into cubemap faces and added to the training set.
- The 2DGS model is refined using a combination of standard photometric loss on original views and a downweighted loss on completed views, alongside geometric regularization against the plane-refined depth maps.

Key Contributions

Layout Anchored Attention Steering: A training-free mechanism that guides panoramic scene completion by anchoring the diffusion model's internal attention to the scene's 3D layout (specifically walls, floors, and ceilings). This ensures unobserved regions are completed as geometric extensions of detected surfaces.
Semantic-Aware Plane Assignment: A pipeline that uses VLM-based reasoning to distinguish between structural layout planes and non-structural objects, enabling selective geometric steering that avoids hallucinating flat textures on furniture.
Panoramic Priors for Sparse-View Reconstruction: Demonstrates that 360° panoramic completion provides superior global context compared to perspective-based generation, leading to higher fidelity novel view synthesis.

Results

Experiments were conducted on three standard indoor benchmarks: Replica, ScanNet++, and Matterport3D, across 3, 6, and 9 input views.

Performance: PanoPlane achieves state-of-the-art performance in PSNR, SSIM, and LPIPS across all datasets and view counts.
- On the Replica dataset with 3 input views, it achieves a +2.71 dB improvement in PSNR over the best baseline.
- On Matterport3D (6 views), it outperforms the strongest baseline (G4Splat) by +2.60 dB.
Efficiency: The method requires no training or fine-tuning of the underlying diffusion model. The additional runtime for the steering mechanism is negligible, with total inference times comparable to other generative baselines.
Ablation: Removing the layout steering mechanism causes the largest performance drop (approx. -1.02 dB), confirming it as the primary driver of geometric consistency. Semantic filtering is also shown to be critical; without it, steering often directs attention to non-layout surfaces, negating benefits.

Significance and Claims

The paper claims that PanoPlane establishes a new paradigm for sparse-view indoor reconstruction by shifting from fragmented perspective generation to holistic 360° completion. By explicitly steering the diffusion model's attention to the scene's underlying planar structure, the method achieves geometrically consistent completions without requiring per-scene fine-tuning.

The authors position this work as a bridge between geometric regularization (for observed regions) and generative completion (for unobserved regions). They note that the method assumes detectable planar structures and may struggle in environments dominated by curved surfaces or heavy clutter that occludes the structural envelope. Future work is suggested to include adaptive steering strengths and extensions to non-planar primitives.