💻 computer science

Rectifying Mask via Entropy for Distractor-Free 3DGS in Ambiguous Scenarios

The paper introduces RefineSplat, a novel framework that leverages entropy-aware adaptive masking and density control to effectively identify and remove ambiguous distractors for high-quality, distractor-free 3D Gaussian Splatting in challenging scenarios, validated by a newly released Ambiguous Wild dataset.

Original authors: Wongi Park, Jiyeon Lim, Minjae Lee, Myeongseok Nam, Seongjun Choi, Jungwoo Kim, Soomok Lee, William J. Beksi, SangHyun Lee

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wongi Park, Jiyeon Lim, Minjae Lee, Myeongseok Nam, Seongjun Choi, Jungwoo Kim, Soomok Lee, William J. Beksi, SangHyun Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, 3D digital replica of a real-world room using a series of photos. You want to be able to walk around this digital room and look at it from any angle. This is what the technology called 3D Gaussian Splatting (3DGS) does. It builds the scene using millions of tiny, fuzzy, 3D "clouds" (called Gaussians) that act like pixels in 3D space.

However, there's a big problem when you take photos in the real world: distractors.

Think of a distractor as an uninvited guest at a party. It could be a person walking through the frame, a car driving by, or a statue that looks exactly like a real person. In a perfect digital reconstruction, you only want the "static" party (the walls, the furniture). You don't want the "uninvited guests" (the moving people or cars) mixed into the 3D model. If you include them, the 3D model gets blurry, ghostly, or warped because the computer gets confused about what is part of the room and what is just passing through.

The Problem: The "Look-Alike" Confusion

Existing methods try to remove these uninvited guests by looking for two things:

Color differences: "If it's a different color than the wall, it's a guest."
Semantic differences: "If it looks like a person, it's a guest."

But the authors of this paper, RefineSplat, point out a major flaw: What if the guest looks exactly like the host?

Imagine a red car parked next to a red brick wall. The computer sees "red" and "red" and thinks, "Oh, that's just part of the wall."
Imagine a mannequin standing next to a real person. The computer sees "human shape" and "human shape" and thinks, "That's just a person standing still."

When the computer can't tell the difference because the colors or shapes are too similar, it accidentally builds the moving guest into the static wall. This creates a messy, blurry 3D model.

The Solution: The "Confusion Meter" (Entropy)

The authors introduce a new framework called RefineSplat. Instead of just asking "Is this a different color?" they ask a smarter question: "How confused is the computer about this part of the image?"

They use a concept called Entropy, which you can think of as a "Confusion Meter."

Low Entropy (Low Confusion): The computer is very sure what it is seeing. "This is definitely a wall."
High Entropy (High Confusion): The computer is torn. "Is this a person? Or a statue? Is this a car? Or a shadow?"

RefineSplat uses this Confusion Meter to find the trouble spots.

The Smart Mask: When the computer is highly confused (high entropy) about a specific object, RefineSplat puts a "mask" over it. It says, "We don't know what this is, so let's treat it as a temporary intruder and ignore it while building the permanent 3D room."
The Smart Cleanup: Once the intruders are masked out, the computer rebuilds the 3D model using only the clear, confident parts of the image.

The "Tidy Up" Process (Density Control)

Building a 3D model is like placing millions of tiny marbles in a box. If you place them randomly, they clump together and look messy.

Old methods tried to fix clumps by looking at how much the color changed (photometric error). But if the colors are similar (like the red car and red wall), they place the marbles in the wrong spots, making the model look blurry.
RefineSplat uses the Confusion Meter to guide where the marbles go. If an area is confusing, it doesn't force the marbles to settle there. Instead, it moves them to the clear, confident areas. This keeps the 3D model sharp and clean, without the "ghosts" of the moving objects.

The New "Tricky Room" Dataset

To prove their method works, the authors realized that existing test sets weren't hard enough. They created a new dataset called "Ambiguous Wild."

Imagine a dataset full of "tricky rooms" where the uninvited guests look exactly like the furniture.
They captured 18 scenes where, for example, a moving car looks just like a parked one, or a person looks just like a statue.
They tested their method on these tricky rooms and showed that while other methods got confused and made blurry models, RefineSplat successfully removed the intruders and kept the room clear.

Summary

In short, RefineSplat is a new way to build 3D digital worlds from photos. It solves the problem of "look-alike" intruders (like a red car against a red wall) by using a Confusion Meter (Entropy). Instead of guessing based on color or shape, it identifies where the computer is unsure, masks out those confusing parts, and builds a clean, sharp 3D model without the unwanted moving objects.

Technical Summary: RefineSplat via Entropy for Distractor-Free 3DGS in Ambiguous Scenarios

Problem Statement

Realistic 3D novel view synthesis (NVS) from diverse, unconstrained real-world scenarios is hindered by the presence of "distractors"—transient elements such as dynamic objects, varying occlusions, or appearance variations. While existing methods based on 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) have made strides, they struggle significantly in ambiguous scenarios.

Current approaches typically rely on three paradigms:

Residual-based masking: Uses photometric error to generate transient masks.
Semantic-based masking: Leverages features from models like DINOv2 to identify distractors.
Heuristic-based masking: Uses extracted masks (e.g., from SAM) conditioned on photometric error.

However, these methods face critical limitations in ambiguous real-world settings where distractors share similar colors or semantic levels with static background elements (e.g., a moving vehicle vs. a parked one, or a person vs. a statue). In such cases:

Photometric errors fail to distinguish transient elements, leading to imprecise masks.
Density control based on photometric gradients misaligns Gaussians, degrading rendering quality.
Semantic features alone cannot resolve instances with high semantic similarity.

Methodology: RefineSplat

The authors propose RefineSplat, a systematic framework designed to construct transient masks and align Gaussians specifically for ambiguous scenarios. The method comprises three core components:

1. Entropy-Aware Adaptive Masking

To capture ambiguous distractors that are difficult to distinguish by color or semantics, RefineSplat integrates Shannon entropy with instance masks.

Instance-Level Analysis: The method utilizes a segmentation model (Grounded SAM) to obtain instance masks. It calculates the average entropy for each instance using semantic features extracted from a frozen DINOv2 model.
Adaptive Thresholding: Instead of using fixed or statistical thresholds (mean/variance) which fail on skewed distributions common in ambiguous scenes, RefineSplat employs an adaptive entropy threshold ( $T_{ent}$ ). This threshold is optimized during training by maximizing the sum of entropy between lower and upper subsets of the instance entropy distribution.
Global Refinement: A global adaptive mask is constructed using an uncertainty map derived from the ratio of uncertainty to entropy, smoothed by a box filter.
Final Mask: The robust final mask ( $M_{final}$ ) is the union of the local adaptive mask and the global mask.

2. Entropy-Aware Density Control

Standard 3DGS density control relies on positional gradients derived from photometric loss, which can misalign Gaussians in ambiguous regions. RefineSplat introduces a novel density control strategy:

Entropy-Driven Gradients: The method derives new Gaussians based on the gradient of an identity loss ( $L_{id}$ ) that incorporates entropy. This allows the system to clone and split Gaussians based on entropy magnitude rather than just photometric error.
Merging Process: To reduce redundancy and artifacts, overlapping Gaussians are merged. The merging criteria consider both the overlap ratio and color similarity, aggregating attributes (position, color, opacity, covariance) in a weighted manner.

3. Consistency Regularization

To rectify potential errors in extracted instance masks, a consistency regularization term ( $L_{cr}$ ) is introduced. This term minimizes the Kullback-Leibler (KL) divergence between the entropy values of pixels within the same segment, ensuring that pixels within a segment represent a consistent identity.

The total optimization loss combines the standard 3DGS loss (weighted by the final mask), the identity loss, and the consistency regularization.

Key Contributions

RefineSplat Framework: The first approach to leverage entropy and instance masks to capture ambiguous distractors in real-world scenarios, addressing the failure modes of prior residual and semantic-based methods.
Entropy-Aware Density Control: A novel strategy that aligns Gaussians using entropy magnitude as a positional gradient, effectively mitigating artifact issues caused by misalignment in ambiguous regions.
Ambiguous Wild Dataset: The creation and release of a new dataset containing 18 challenging scenes (11 outdoor, 7 indoor) where distractors and static elements are difficult to distinguish due to color or semantic similarities. This dataset is designed to rigorously evaluate methods in ambiguous conditions.
State-of-the-Art Performance: Extensive experiments demonstrate that RefineSplat outperforms existing methods across multiple datasets, achieving distractor-free novel view synthesis with superior quantitative metrics (PSNR, SSIM, LPIPS) and memory efficiency.

Experimental Results

The authors evaluated RefineSplat on the Ambiguous Wild dataset, NeRF On-the-go, Photo Tourism, and Drone Imagery datasets.

Quantitative Performance: RefineSplat consistently achieved the highest scores in PSNR, SSIM, and lowest LPIPS compared to baselines like WildGaussians, RobustSplat, DroneSplat, and GS-W. For instance, on the Ambiguous Wild dataset, it achieved a PSNR of 23.83 (vs. 20.52 for the baseline) and an LPIPS of 0.149.
Qualitative Results: Visualizations show that RefineSplat successfully removes ambiguous distractors (e.g., people blending into statues, vehicles with similar colors) where other methods either fail to mask them or introduce artifacts.
Efficiency: The entropy-aware density control and merging process resulted in significant memory efficiency, reducing the number of Gaussians and memory usage compared to baselines while maintaining visual quality.
Ablation Studies: Experiments confirmed the necessity of each module (Entropy-aware masking, Density control, and Consistency regularization). The adaptive entropy threshold ( $T_{ent}$ ) was shown to be more robust to skewed distributions than fixed or statistical thresholds.

Significance and Claims

The paper claims that RefineSplat is the first work to pave the way for novel view synthesis in ambiguous real-world scenarios. Its significance lies in:

Addressing a Fundamental Gap: It solves the specific problem of distinguishing transient elements from static scenes when they share visual or semantic similarities, a limitation that has hindered prior unconstrained NVS methods.
Novel Mechanism: By formulating entropy-aware methods for both masking and density control, it moves beyond simple photometric or semantic heuristics.
Community Resource: The release of the Ambiguous Wild dataset provides a standardized benchmark for future research in handling ambiguous distractors, a scenario previously underrepresented in the literature.

The authors acknowledge limitations, noting that performance depends on the underlying vision foundation models and that generalization to extremely sparse views remains challenging. They suggest future work could integrate diffusion models to inpaint sparse regions or extend the method to large-scale dynamic scenes.