💻 computer science

Learning Stable Canonical Worlds for Novel View Synthesis and Beyond

This paper introduces CanonicalGS, a feed-forward Gaussian splatting pipeline that aggregates multi-view observations into a stable, scene-centric canonical latent world using uncertainty-aware fusion, thereby improving novel view synthesis quality and downstream perception accuracy by suppressing noisy or redundant evidence.

Original authors: Xiaoyu Xu, Jian Zou, Sheyang Tang, Zhihua Wang, Jing Liao, Kede Ma

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaoyu Xu, Jian Zou, Sheyang Tang, Zhihua Wang, Jing Liao, Kede Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a room, but instead of walking around it yourself, you are given a stack of photos taken from different angles.

The Problem: The "Noisy Crowd" Effect
Current methods for turning these photos into 3D models act a bit like a chaotic crowd of people trying to describe the same object. If you ask one person, they might say, "It's a red chair." If you ask another, they might say, "No, it's a blue chair," or "It's a chair with a scratch."

Existing AI methods often just listen to everyone and try to blend these conflicting stories together. As you add more photos (more people), the AI doesn't necessarily get a clearer picture; instead, it gets confused by the noise, the contradictions, and the redundant information. The more photos you add, the messier the final 3D model becomes, full of "ghosts" or blurry artifacts.

The Solution: The "Smart Editor" (CanonicalGS)
The authors of this paper, CanonicalGS, propose a new way to handle this. Instead of letting every photo shout its own story, they introduce a "Smart Editor" that works in three steps:

The Detective Work (View-Centric Evidence):
First, the system looks at each photo individually. It doesn't just look at the colors; it acts like a detective checking the facts. It asks:
- "How deep is this object?" (Depth)
- "Does this photo look blurry or unclear?" (Uncertainty)
- "Is this feature sharp and clear?" (Reliability)
  It assigns a "trust score" to every part of every photo. If a photo is blurry or the depth is confusing, it gets a low trust score.
The Roundtable Meeting (Scene-Centric Aggregation):
This is the magic step. Instead of keeping the photos separate, the system projects all the information into a single, shared "virtual room" (a canonical world).
- Imagine a whiteboard where everyone writes their observations.
- If the "Smart Editor" sees that Photo A and Photo B both agree on a chair's location and shape, and both have high trust scores, they reinforce each other. The chair becomes clearer and more solid.
- If Photo C says the chair is in a different spot, but Photo C has a low trust score (maybe it was blurry), the Editor ignores it.
- The Goal: To filter out the noise and only let the reliable, agreeing evidence build the final model. The more good photos you add, the stronger and clearer the model gets, rather than messier.
The Final Sculpture (GP Decoding):
Once the "virtual room" is filled with only the reliable, agreed-upon information, the system finally builds the 3D model (using something called "Gaussian Splatting," which is like painting the scene with millions of tiny, fuzzy 3D dots). Because the input data was cleaned up first, the final sculpture is sharp, stable, and accurate.

Why This Matters (The Results)
The paper claims this approach is a game-changer for two main reasons:

Better Pictures: When they tested it, adding more photos actually made the new views look better. While other methods got worse or stopped improving after a few photos, CanonicalGS kept getting sharper. They saw a significant improvement in image quality (up to 2.5 dB better).
Smarter Understanding: Because the system built a "clean" and "stable" 3D world, it didn't just look good; it understood the scene better. When they used this model to identify objects (like "this is a sofa," "that is a table"), it was 11% more accurate than other methods.

In Summary
Think of CanonicalGS as a filter that turns a chaotic pile of conflicting eyewitness accounts into a single, undeniable truth. It doesn't just stack photos on top of each other; it listens to the reliable ones, ignores the confused ones, and builds a stable, high-quality 3D world that gets better the more evidence you provide.

Technical Summary: Learning Stable Canonical Worlds for Novel View Synthesis and Beyond

Problem Statement

Current Feed-Forward Gaussian Splatting (FFGS) methods facilitate real-time novel view synthesis but suffer from a "rendering-first" bias. These approaches typically optimize multi-view evidence to synthesize target views directly, often resulting in view-dependent predictions. As the number of input views increases, these methods tend to accumulate noisy, redundant, or conflicting evidence rather than converging to a stable, shared scene representation. Consequently, additional views may degrade performance or introduce artifacts (e.g., ghosting, inconsistent geometry) instead of refining the scene model. The paper argues that existing methods fail to perform "canonization"—the transformation of cluttered, view-centric observations into a coherent, scene-centric internal model—a concept rooted in Marr's theory of vision.

Methodology: CanonicalGS

The authors propose CanonicalGS, a feed-forward pipeline designed to map multi-view observations into a stable, scene-centric latent world before decoding 3D Gaussians. The method follows a 2D → 2.5D → 3D pipeline:

1. View-Centric Evidence Formation

For each input view, the system extracts dense visual features, depth, and uncertainty estimates independently:

Feature Extraction: A pretrained DINO-v2 backbone extracts patch-level features, which are upsampled to image resolution to form a dense feature map ( $F_{view}$ ).
Uncertainty-Aware Depth Estimation: Using a cost-volume depth module inspired by plane-sweep stereo, the model computes a discrete depth probability volume. This yields a depth map ( $D$ ) and a positional uncertainty map ( $U_{pos}$ ) derived from the variance of depth hypotheses.
Appearance Uncertainty: An additional appearance uncertainty map ( $U_{app}$ ) is predicted from semantic features via an MLP.
Reliability Map: These uncertainty sources are combined to generate a per-pixel reliability map ( $R_{view}$ ), where high reliability indicates geometric support and visual certainty.

2. Scene-Centric Evidence Aggregation

The core innovation lies in aggregating evidence in a shared 3D voxel lattice ( $\mathcal{V}$ ) rather than in image space or after Gaussian decoding:

Rasterization: Lifted 3D points from all views are rasterized onto a fixed voxel grid.
Reliability-Guided Aggregation: Within each voxel, the system identifies a "local representative" (the observation with the highest reliability). Other observations are weighted based on their reliability and their feature agreement with this representative.
- Observations that are reliable and consistent with the local representative contribute strongly.
- Uncertain or inconsistent observations are suppressed.
Output Fields: This process produces two scene-centric fields:
- $R_{scene}$ : A reliability field recording accumulated support.
- $F_{scene}$ : A feature field storing reliability-weighted scene descriptors.

3. GP Decoding

A decoder maps the latent scene fields ( $F_{scene}, R_{scene}$ ) into a renderable set of 3D Gaussians ( $G$ ):

Opacity Control: The opacity ( $\alpha$ ) of each Gaussian is derived from the accumulated scene reliability ( $R_{scene}$ ) via a non-decreasing function. This ensures that as more reliable evidence accumulates, the opacity increases or saturates, rather than fluctuating.
Geometry and Appearance: The remaining attributes (position offset, covariance, spherical harmonics) are decoded from the feature field ( $F_{scene}$ ). The position is anchored to the representative lifted point and refined by a learned offset.

Key Contributions

Canonization-Oriented FFGS: The paper reframes FFGS from a rendering pipeline to a representation learning algorithm. It explicitly separates evidence consolidation (in scene space) from primitive prediction (Gaussian decoding).
Uncertainty-Aware Fusion: Unlike prior aggregation methods that may treat all views equally or merge after decoding, CanonicalGS uses uncertainty as a first-class signal to weight evidence before aggregation, suppressing noise and redundancy.
Stable Scene Representation: By enforcing a non-decreasing opacity constraint based on accumulated reliability, the method ensures that additional views refine the scene representation rather than destabilizing it.
Downstream Transferability: The architecture naturally produces features suitable for downstream tasks (e.g., semantic segmentation) without requiring auxiliary perception heads, as the scene representation is built on consolidated, reliable evidence.

Experimental Results

The authors evaluate CanonicalGS on RealEstate10K (RE10K) and DL3DV datasets, comparing against state-of-the-art FFGS methods (MVSplat, DepthSplat, FreeSplat, ZPressor) and Gaussian-space merging variants.

Novel View Synthesis: CanonicalGS achieves significant improvements in Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and LPIPS. Notably, while baseline methods often degrade or plateau as input views increase (2 to 8 views), CanonicalGS shows consistent improvement, achieving up to a 2.5 dB gain in PSNR on RE10K.
Representation Stability: Qualitative and quantitative analyses show that CanonicalGS features become more stable and converge toward a reference (12-view) as more inputs are added, whereas baselines exhibit sensitivity to view-set changes.
Downstream Perception: When used for semantic segmentation via a linear probe, CanonicalGS yields an 11% gain in accuracy compared to baselines, demonstrating that the learned latent world preserves semantic structure better.
Robustness: Under Gaussian noise perturbations to depth estimates, CanonicalGS maintains higher PSNR and SSIM with smaller degradation compared to depth-conditioned baselines, validating the effectiveness of reliability-guided aggregation.

Significance and Claims

The paper claims that CanonicalGS successfully bridges classical vision theory (Marr's hierarchy) with modern 3D Gaussian representations. Its primary significance lies in demonstrating that additional views should strengthen a shared scene representation rather than introducing view-conditioned clutter.

The authors position CanonicalGS not merely as a rendering tool but as a representation learning algorithm that enables:

Scalability: Performance improves with more input views, unlike current rendering-first approaches.
Robustness: The system is less sensitive to noisy depth estimates and occlusions due to uncertainty-aware fusion.
Generalization: The stable, scene-centric features transfer effectively to downstream visual perception tasks like semantic segmentation.

The paper concludes modestly, acknowledging limitations such as dependence on the quality of geometric inputs (camera poses and depth estimates) and the fixed structure of the voxel lattice. It suggests future work in joint canonicalization (refining depth and pose together) and richer uncertainty modeling, but emphasizes that the core contribution is the shift toward evidence consolidation in a shared scene space.