💻 computer science

SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis

SplatGuide achieves state-of-the-art pose-free novel view synthesis by unifying rendered images, per-Gaussian visibility voting maps, and reconstruction tokens from a single 3D Gaussian scene to provide comprehensive geometric and feature guidance for multi-view diffusion.

Original authors: Yejun Zhang, Zihan Wang, Xu Ji, Yihao Wang, Yuxin Hou, Junyuan Fang, Juho-Matti Kilpeläinen, Arno Solin, Hamed Rezazadegan Tavakoli, Esa Rahtu, Juho Kannala

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yejun Zhang, Zihan Wang, Xu Ji, Yihao Wang, Yuxin Hou, Junyuan Fang, Juho-Matti Kilpeläinen, Arno Solin, Hamed Rezazadegan Tavakoli, Esa Rahtu, Juho Kannala

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine holding a smartphone and snapping a few casual photos of a room as you walk through it. You have no special equipment, no measured distances, and no record of where the camera was pointing. Now, imagine asking a computer to generate a brand-new, photorealistic image of that same room from a spot where you never stood, perhaps looking around a corner you never saw. This is the challenge of creating new views from unposed images. For years, scientists have relied on two separate tools to tackle this. One tool is excellent at understanding the shape and layout of a scene, building a rough 3D map from flat pictures, but it struggles to invent details it has never seen. The other tool is a powerful image generator that can create stunning new pictures, but it usually demands precise instructions about exactly where the camera was, instructions that are rarely available in casual snapshots. When researchers tried to combine these tools, they often found that the connection between the 3D map and the image generator was weak, leaving the system to guess at the geometry or ignore the map entirely.

A team of researchers has now bridged this gap with a new method called SplatGuide. Instead of treating the 3D reconstruction and the image generation as separate steps, they designed a system that reuses the same 3D model three different times to guide the process. First, the system takes the unposed photos and builds a 3D scene made of millions of tiny, colored clouds, known as Gaussians. This is a standard way to represent 3D space, but the researchers noticed that existing methods threw away most of the information this model held. SplatGuide keeps every piece of data. It uses the 3D model to paint a rough, geometric picture of what the new view should look like, providing a solid anchor for the image generator. It then uses the model to figure out which of the original photos are actually visible from the new angle, ignoring those that are blocked by walls or furniture, ensuring the system only looks at the most helpful references. Finally, it extracts high-level features from the 3D model to tell the generator about the overall style and structure of the room, not just the pixels it can see.

The results show that this approach works remarkably well. By feeding the image generator with these three distinct signals—the rough geometric picture, the smart selection of reference photos, and the structural features—the system creates new views that are sharper and more accurate than previous methods. In tests on standard datasets, the method produced images that were so convincing they surpassed even systems that were given perfect, ground-truth camera positions, provided enough input photos were available. The researchers found that the most significant improvement came from how they selected which reference photos to use. Instead of just picking photos that were taken from nearby angles, their system looked at the 3D map to see which photos actually showed the parts of the scene visible from the new spot. This prevented the system from wasting time on redundant images or getting confused by objects blocking the view.

What makes this work particularly robust is its flexibility. The system does not depend on a single specific way of building the 3D map; it can swap in different reconstruction tools without needing to be retrained, meaning it will automatically improve as those underlying tools get better. The researchers also tested the system on scenes it had never seen before, including large outdoor environments and complex indoor spaces, and it maintained its high quality. While the system is not perfect and can struggle if the original photos are too blurry or if the scene contains moving objects, it represents a major step forward. It proves that by carefully reusing the information already present in a 3D reconstruction, computers can learn to see the world in three dimensions and imagine new perspectives with a level of clarity that was previously out of reach.

Technical Summary: SplatGuide

Problem Statement

Pose-free Novel View Synthesis (NVS) aims to generate photorealistic images from unposed, casual snapshots without camera calibration or pose annotations. This task requires two distinct capabilities that currently exist in separate paradigms:

3D Geometric Understanding: Feed-forward reconstruction models (e.g., DUSt3R, VGGT) can recover scene geometry and camera poses in a single pass but are limited to interpolating between observed viewpoints and cannot hallucinate content in unobserved regions.
Content Synthesis: Multi-view diffusion models can generate high-fidelity novel views but typically rely on accurate, pre-computed ground-truth poses, which are unavailable in pose-free settings.

Existing attempts to combine these approaches suffer from a structural information disconnect. Current pipelines extract at most one signal from the 3D reconstruction (e.g., estimated poses, rendered images, or latent features) and discard the rest. Furthermore, reference view selection in these pipelines often relies on heuristics like pose distance or temporal recency, which ignore occlusion and fail under tight context budgets. This disconnect leaves valuable geometric, visibility, and feature information unused, creating a bottleneck in generation quality.

Methodology: SplatGuide

SplatGuide addresses this disconnect by reusing a single 3D Gaussian Splatting (3DGS) scene, generated by a feed-forward reconstruction model, across three complementary roles to guide a multi-view diffusion generator. The pipeline consists of three stages:

1. Reconstruction

A feed-forward backbone (e.g., WorldMirror) processes $N$ unposed input images to produce:

Estimated camera poses.
A 3DGS scene $\mathcal{G}$ where each Gaussian carries a source-view index indicating which input image it originated from.
Reconstruction tokens: Camera tokens and register tokens extracted from the backbone's final layer, aggregating non-local scene information.

2. Visibility-Aware View Selection

Instead of selecting reference views based on pose distance, SplatGuide leverages the per-Gaussian source-view indices to perform occlusion-aware selection:

Index Map Rendering: The 3DGS scene is rendered into the target view using a "first-hit" depth test. Each pixel is assigned the color of the source view of the nearest Gaussian.
Voting: A visibility score $S(k)$ is computed for each candidate view $k$ by counting the number of target pixels dominated by geometry from that view.
Refinement: The selection is refined by DeDup (removing spatially redundant views) and PoseAug (ensuring isolated targets have nearby context), creating a diverse and informative context set within the diffusion model's budget.

3. Generation

The diffusion model (based on SEVA) synthesizes the target view using three conditioning signals derived from the single reconstruction pass:

Geometric Conditioning (Pixel-level): Rendered images of the target and reference poses are concatenated with the noisy latent state. This provides pixel-aligned geometric anchoring.
Feature Conditioning (Token-level): Camera and register tokens are injected via cross-attention layers, supplying global scene context and texture statistics that rendered images alone cannot convey.
Context Views: The selected reference views (RGB and Plücker coordinates) are processed alongside the target inputs.

The reconstruction backbone and VAE remain frozen; only the diffusion model is trained.

Key Contributions

Identification of the Information Disconnect: The paper identifies that the primary bottleneck in pose-free NVS is not the lack of reconstruction capability, but the failure of existing pipelines to utilize the full spectrum of signals (geometry, visibility, features) available from a single 3DGS reconstruction.
SplatGuide Framework: A novel architecture that bridges rendered images, per-Gaussian visibility, and reconstruction tokens from a single 3DGS scene to the diffusion generator, eliminating the need for additional 3D primitives or retraining the backbone.
Visibility-Aware View Selector: A low-cost selector that renders per-Gaussian source-view indices into a voting map. This method outperforms pose-based and recency-based strategies, particularly when the context budget is tight, by explicitly handling occlusion.
Modularity and Zero-Shot Transfer: The framework demonstrates that the reconstruction backbone can be substituted zero-shot (e.g., swapping WorldMirror for AnySplat) without retraining the diffusion model, allowing the system to immediately benefit from advances in feed-forward reconstruction.

Experimental Results

SplatGuide was evaluated on four benchmarks: RealEstate10K, DL3DV, Tanks-and-Temples, and Mip-NeRF 360.

State-of-the-Art Performance: SplatGuide achieves the best results among pose-free methods across all datasets. On RealEstate10K with 9 input views, it surpasses the ground-truth-pose baseline (SEVA), achieving a PSNR of 30.00 compared to 29.63.
Impact of Components:
- View Selection: In isolation, the visibility-aware selector improves PSNR by up to 3.9 dB over camera-distance baselines in sparse regimes (6 views).
- Geometric Conditioning: Adding rendered images to the baseline yields a +0.66 dB gain. Adding tokens further improves quality, with the full method achieving a 16% relative reduction in LPIPS compared to a pose-only baseline.
Generalization: The method maintains superior performance on out-of-domain datasets (Mip-NeRF 360, Tanks-and-Temples) without domain-specific tuning.
Efficiency: The view selection process adds negligible latency (<0.01% of total inference time) as it reuses the 3DGS rasterization already required for geometric conditioning.

Significance and Claims

The paper claims that SplatGuide fundamentally shifts the paradigm of pose-free NVS by treating the 3D reconstruction not merely as a pose estimator or a coarse geometry prior, but as a unified interface for multiple conditioning signals.

Closing the Pose Gap: By effectively utilizing the reconstructed geometry, SplatGuide compensates for the inaccuracies of predicted poses, even surpassing methods that rely on ground-truth poses when sufficient input views are available.
First-Order Design Decision: The authors assert that visibility-aware view selection is a critical, first-order design decision for scalable generation, especially in practical scenarios with limited context budgets.
Modular Advancement: The decoupling of reconstruction and generation allows the framework to act as a "plug-and-play" system where improvements in feed-forward reconstruction models directly translate to higher synthesis quality without retraining the generative model.

The paper concludes that the limitations of the current approach are correlated with the quality of the underlying feed-forward reconstruction; if the reconstruction fails (e.g., on textureless surfaces or dynamic content), all three conditioning signals degrade simultaneously. However, within its operational scope, the method establishes a new state-of-the-art for pose-free novel view synthesis.