💻 computer science

From Uncertainty to Stability and Fidelity: Guiding Sparse-View 3D Gaussian Splatting with Fisher Information

This paper proposes a novel sparse-view 3D Gaussian Splatting method that leverages Fisher Information to actively guide stereo augmentation and adaptively regulate uncertainty-aware regularization, thereby mitigating overfitting and significantly improving rendering stability and fidelity.

Original authors: Junbao Zhou, Qingshan Xu, Yuan Zhou, Xiaolong Shen, Beier Zhu, Kesen Zhao, Yiming Zeng, Chen Bai, Cheng Lu, Hanwang Zhang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junbao Zhou, Qingshan Xu, Yuan Zhou, Xiaolong Shen, Beier Zhu, Kesen Zhao, Yiming Zeng, Chen Bai, Cheng Lu, Hanwang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a room, but you only have a few blurry photos of it instead of a full set of pictures from every angle. This is the challenge of Sparse-View 3D Gaussian Splatting (3DGS).

In the world of computer graphics, "3DGS" is a technique that builds a scene out of millions of tiny, fuzzy balls (Gaussians) that float in space. When you look at the scene from a new angle, the computer blends these balls together to create a realistic image.

The problem is: If you only give the computer a few photos, it gets confused. It tries too hard to memorize those few photos (a problem called "overfitting"), resulting in a model that looks great from the angles it saw but turns into a blurry, glitchy mess from any new angle.

The authors of this paper propose a new way to teach the computer how to learn from these few photos without getting confused. They use a mathematical tool called Fisher Information as a "smart guide" to fix two main problems.

Here is how they do it, explained with simple analogies:

1. The Problem: "Hollow" and "Random" Guessing

Previous methods tried to help the computer by creating fake training photos (called "pseudo ground truths") using depth maps (like a 3D radar scan).

The Old Way: Imagine a student trying to learn a map by looking at just one photo and guessing what the rest of the map looks like. They might guess wrong, leaving "hollow" gaps where buildings should be. Also, they might randomly pick a spot to guess from, which is chaotic and unreliable.
The Paper's Solution (Stereo Augmentation with Fisher Information): Instead of guessing randomly, the computer acts like a smart detective. It uses Fisher Information to ask: "Which of the few photos I already have will teach me the most about this new angle?"
- It picks the most informative photos (the ones that give the best clues).
- It combines clues from multiple photos to build a high-quality "fake" photo to study.
- Result: The computer stops guessing randomly and starts learning from the best possible examples, filling in the "hollow" gaps accurately.

2. The Problem: "Blind" Pruning

To stop the computer from memorizing the few photos too strictly, previous methods used a technique called Dropout. This is like a teacher randomly erasing parts of a student's homework to force them to think harder.

The Old Way: The teacher erases parts of the homework randomly.
- The Bad Part: Sometimes, the teacher erases a part the student already understands perfectly (over-optimized).
- The Worse Part: Sometimes, the teacher leaves alone the parts the student doesn't understand yet (under-optimized).
- This creates a mess: The student gets confused because they are being tested on things they know, but ignored on things they need to learn.
The Paper's Solution (Uncertainty-Aware Regularization): The teacher now uses a smart scorecard (Fisher Information) to check how confident the student is about each specific piece of the puzzle.
- If the student is too confident (over-optimized): The teacher gently removes that part of the homework to force them to double-check their work.
- If the student is unsure (under-optimized): The teacher protects that part, letting them keep practicing until they get it right.
- The "Soft" Touch: Instead of completely deleting a part of the homework (which can cause the student to panic and guess wildly), the teacher just dims it slightly. This keeps the student's focus on the background details without overwhelming them.

The Big Picture

Think of the whole process like training a chef with only three recipes:

Old Method: The chef tries to guess the taste of a new dish by randomly mixing ingredients from the three recipes. The result is often a weird, salty mess.
New Method (This Paper):
- The chef uses a smart guide to figure out which of the three recipes provides the best clues for the new dish (Stereo Augmentation).
- The chef then practices, but a smart coach watches closely. If the chef is already perfect at chopping onions, the coach makes them stop chopping to focus on the sauce. If the chef is struggling with the sauce, the coach lets them keep practicing without interruption (Uncertainty-Aware Regularization).

The Result:
By using this "smart guide" (Fisher Information) to choose the best training examples and to decide exactly what to practice, the computer builds a 3D model that looks incredibly realistic, even when it only had a few photos to start with. The paper proves this works better than any previous method on standard test datasets.

Technical Summary: Guiding Sparse-View 3D Gaussian Splatting with Fisher Information

Problem Statement

3D Gaussian Splatting (3DGS) has established itself as a leading technique for novel view synthesis (NVS), offering real-time rendering speeds and high fidelity compared to Neural Radiance Fields (NeRF). However, 3DGS relies heavily on dense input views to achieve satisfactory results. In sparse-view scenarios (e.g., 3 to 9 input images), the model suffers from severe overfitting due to insufficient geometric and appearance information. This leads to noticeable artifacts, degraded rendering quality, and poor generalization to unseen viewpoints.

Existing approaches to mitigate this issue generally fall into two categories:

Prior-based Augmentation: Using pre-trained models (e.g., depth estimators) to generate pseudo ground truths. However, these methods often rely on random sampling of viewpoints or warping single training views, which fails to fully exploit stereo geometric priors and introduces "hollow" areas in the augmented data.
Regularization Techniques: Methods like Dropout are used to prevent overfitting. However, traditional Dropout applies a uniform removal probability to all Gaussians. This lack of selectivity can inadvertently remove under-optimized Gaussians (e.g., those representing occluded regions) while preserving over-optimized ones, leading to instability and visual artifacts like over-compensation.

Both approaches suffer from compounded randomness, introducing uncertainty and instability that limit the fidelity of the final synthesis.

Methodology

The authors propose a novel framework that utilizes Fisher Information to quantitatively guide both data augmentation and regularization, moving from random heuristics to principled selection. The method consists of two core components:

1. Stereo Augmentation with Fisher Information (SAFI)

To address the issues of "Hollow and Random Augmentation," SAFI actively selects the most informative training views to curate high-quality pseudo ground truths.

Viewpoint Sampling: A novel viewpoint is sampled by perturbing an existing training camera pose (random noise on position and orientation).
Informative View Selection: Instead of randomly selecting support views or using simple Euclidean distance, the method employs Information Gain (IG) derived from Fisher Information. It calculates the reduction in model uncertainty when a candidate view is added.
- The IG is approximated using the trace of the Hessian matrix (derived from Fisher Information) to identify views that maximize visual similarity and geometric constraints relative to the novel viewpoint.
- A set of support views is selected based on a threshold relative to the minimum IG, ensuring multi-view geometric consistency.
Pseudo Ground Truth Curation: Using a pre-trained depth model (Depth-Anything-3), the selected views are unprojected into a 3D point cloud and rasterized from the novel viewpoint to create a high-quality pseudo ground truth image. This dataset is then integrated into the training set.

2. Uncertainty-aware Regularization (UAR)

To address "Regularization without Selectivity," UAR adaptively adjusts the dropout probability for each 3D Gaussian based on its optimization status.

Uncertainty Quantification: The method calculates an uncertainty score ( $U_i$ ) for each Gaussian $G_i$ using the Fisher Information matrix. A higher score indicates the Gaussian is under-optimized (perturbing it has little impact on the rendering), while a lower score indicates it is over-optimized.
Ranking-based Selective Dropout: Gaussians are sorted by uncertainty. The method prioritizes the removal of over-optimized Gaussians (low uncertainty) while preserving under-optimized ones (high uncertainty). This ensures that occluded or under-trained regions receive sufficient optimization.
Soft-scale Dropout: Unlike traditional Dropout which completely zeros out the opacity of selected Gaussians (causing rays to penetrate to occluded regions and forcing over-compensation), UAR employs a soft-scale strategy. Selected Gaussians have their opacity scaled down by a decay factor $\lambda \in (0, 1)$ rather than being removed entirely. This maintains a soft presence in the rasterization, protecting occluded Gaussians from erroneous updates.

Key Contributions

SAFI Strategy: A novel augmentation strategy that leverages Fisher Information to select the most informative views for curating pseudo ground truths, thereby maximizing the utility of geometric priors and reducing randomness in data augmentation.
UAR Strategy: An uncertainty-aware regularization mechanism that quantitatively measures the optimization status of each 3D Gaussian and adaptively adjusts dropout probabilities, leading to more stable optimization and higher rendering fidelity.
Soft-scale Dropout: The integration of a soft-scale dropout mechanism to prevent the over-compensation artifacts associated with hard removal strategies in sparse-view settings.

Experimental Results

The method was evaluated on three standard benchmarks: LLFF, Mip-NeRF 360, and DTU, under sparse-view settings (3, 6, and 9 views for LLFF/DTU; 12 and 24 views for Mip-NeRF 360).

Quantitative Performance: The proposed method achieves state-of-the-art (SOTA) performance across nearly all metrics (PSNR, SSIM, LPIPS).
- On LLFF (3-view), it surpasses the previous best method (DropGaussian) by 0.36 dB in PSNR.
- On DTU (3-view), it achieves 22.44 dB PSNR, significantly outperforming baselines like 3DGS (10.99 dB) and DropGaussian (21.44 dB).
- On Mip-NeRF 360 (12-view), it improves PSNR by 0.35 dB over the previous SOTA.
Stability: The method significantly reduces the standard deviation of PSNR across multiple runs (e.g., from 0.084 to 0.027 in augmentation comparisons), demonstrating improved optimization stability.
Qualitative Results: Visual comparisons show that the method produces novel views with fewer artifacts (e.g., hollow areas, floating artifacts) and better structural fidelity compared to baselines like CoR-GS and DropGaussian.

Significance and Claims

The paper claims to establish a new state-of-the-art in sparse-view novel view synthesis by addressing the fundamental issues of randomness and lack of selectivity in current augmentation and regularization techniques. By grounding the selection of pseudo ground truths and the application of regularization in Fisher Information, the authors provide a principled approach to:

Reduce the uncertainty and instability inherent in random sampling and uniform dropout.
Fully utilize geometric priors through multi-view stereo constraints.
Differentiate between over-optimized and under-optimized Gaussians to prevent visual artifacts like over-compensation.

The authors conclude that their method effectively mitigates overfitting in sparse-view 3DGS, resulting in superior rendering fidelity and stable optimization without requiring additional dense input data.