💻 computer science

GlowGS: Generative Semantic Feature Learning for 3D Gaussian Splatting in Nighttime Glow Scenes

GlowGS addresses the limitations of existing 3D Gaussian Splatting methods in nighttime glow scenes by leveraging a diffusion model and a Vision Foundation Model to generate semantic features as implicit structural cues, enabling robust novel-view synthesis without ground truth supervision.

Original authors: Beibei Lin, Xiao Cao, Jingyuan Guo, Robby T. Tan

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beibei Lin, Xiao Cao, Jingyuan Guo, Robby T. Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a city at night using only a few photos. In the daytime, this is easy because buildings have clear edges, windows, and textures that act like "fingerprint clues" for your computer to figure out where everything is in 3D space.

But at night, things get tricky. The city is dark, and the only things you can see are glowing streetlights, neon signs, and the soft, fuzzy halos (glow) around them. These glowing areas have no edges or textures; they are just smooth, blurry blobs of light.

The Problem: The Computer Gets Lost in the Dark
Existing 3D modeling tools (called 3D Gaussian Splatting) are like master builders who rely on those "fingerprint clues" (edges and textures). When they try to build a 3D model of a night scene, they get confused by the glowing blobs. Because there are no edges to grab onto, the computer tries to guess, and it often makes mistakes. It might create duplicate lights, weird bright spots in the dark, or floating artifacts that look like digital ghosts. It's like trying to assemble a puzzle where half the pieces are just blank white paper.

The Solution: GlowGS (The "Imagination" Builder)
The authors of this paper, led by Beibei Lin, created a new method called GlowGS. Instead of giving up on the dark, they taught the computer how to "imagine" the missing clues. They did this using two main tricks:

1. The "Dreaming" Phase (Semantic Feature Generation)

Since the computer can't see the edges in the dark, the authors asked a "Dream Machine" (a type of AI called a Diffusion Model) to imagine what the scene would look like if the camera moved slightly.

The Analogy: Imagine you have a photo of a glowing streetlamp. You ask an artist to draw what the lamp looks like from a slightly different angle, even though you don't know exactly where that angle is. The artist draws a few variations.
The Quality Check: Sometimes the artist might draw something weird or wrong. So, the authors use a "Super-Observer" (a Vision Foundation Model like DINO or CLIP) to check the drawings. This observer doesn't care about the exact colors; it cares about the meaning and structure of the image. It asks, "Does this new drawing still look like the same streetlamp?" If the answer is yes, it keeps the drawing. If it's too weird, it throws it away and asks for a new one.
The Result: They build a "Library of Clues" (a Semantic Feature Bank) containing these high-quality, imagined views. These views act as a cheat sheet, telling the 3D builder what the glowing areas should look like structurally, even if the original photos were blurry.

2. The "Matching" Phase (Novel-View Semantic Learning)

Now, the 3D builder starts constructing the scene. Usually, it only learns from the original photos. But with GlowGS, it also looks at the "Library of Clues" it just built.

The Analogy: Imagine the 3D builder is painting a new view of the streetlamp. It looks at its "Library of Clues" and finds a drawing that looks very similar to what it's trying to paint. It then says, "Okay, I'll make my painting look more like that high-quality drawing."
The Magic: It doesn't need to know the exact camera angle of the drawing. It just matches the patterns and structures. By constantly comparing its work to the best examples in the library, the builder stops making mistakes. The glowing lights become smooth and realistic, and the weird floating artifacts disappear.

The New Dataset: NightGlow

The authors realized no one had a good collection of night photos with strong glowing lights to test this on. So, they created a new dataset called NightGlow. It's like a special training camp with 18 different night scenes, all featuring those tricky glowing effects, specifically designed to test if their new method works.

The Results

When they tested GlowGS against other top methods:

Old methods produced night scenes with glitchy lights and fuzzy, unnatural blobs.
GlowGS produced scenes where the lights looked smooth, realistic, and consistent, just like a real night photo.
They measured this with a score (PSNR), and GlowGS beat the previous best method by a significant margin (about 1.78 points), proving that "imagining" the missing structural clues works better than just guessing.

In Summary:
GlowGS is a clever workaround for building 3D night scenes. Instead of struggling to find edges in the dark, it uses AI to "dream up" what the scene should look like, checks those dreams for quality, and then uses those dreams as a guide to build a perfect, artifact-free 3D model. It's like giving a blindfolded builder a set of perfect blueprints so they can build a house even when they can't see the bricks.

Technical Summary: GlowGS

Problem Statement

Existing 3D Gaussian Splatting (3DGS) methods excel at rendering high-quality novel views in clear-day scenes but struggle significantly with nighttime environments, particularly in regions exhibiting "glow" (e.g., light sources, streetlamps, neon signs). These glow regions lack distinct structural features such as textures and edges, which are critical cues for splatting-based reconstruction. Consequently, while 3DGS can model these areas in training views, it fails to generalize to novel views, resulting in artifacts like duplicated light sources and unnatural bright spots in dark areas. The core challenge is the absence of reliable geometric structures in glow regions to guide the optimization of 3D Gaussians.

Methodology: GlowGS

To address the lack of structural cues in nighttime glow scenes, the authors propose GlowGS, a method that leverages a diffusion model and a Vision Foundation Model (VFM) to compensate for missing information. The framework operates through two novel components: Semantic Feature Generation and Novel-View Semantic Learning.

1. Semantic Feature Generation

This component aims to synthesize high-quality semantic features that serve as implicit structural cues for novel views, even when camera poses are unknown.

Image-to-Video Generation: The method employs image-to-video diffusion models (e.g., Pika, PromeAI) to synthesize novel views from training images. These models generate video sequences with unknown camera poses, effectively capturing multi-view glow information.
VFM-Based Verification: Since generative models can introduce hallucinations and inconsistencies, a VFM (such as DINO or CLIP) is used to verify the quality of generated views. The VFM extracts features from both the original training view and the generated novel views. The $L_2$ distance between these features is computed; if the distance exceeds a specific threshold, the view is deemed low-quality and regenerated.
Feature Extraction: Once high-quality novel views are identified, the VFM extracts robust, patch-level semantic features. These features are aggregated into a Semantic Feature Bank. This bank serves as a repository of structural cues derived from the glow regions, independent of precise camera pose information.

2. Novel-View Semantic Learning

This component enables 3DGS to optimize novel views without requiring ground-truth supervision or known camera poses for the generated views.

Mechanism: During training, 3DGS renders novel views. The VFM extracts semantic features from these rendered views.
Retrieval and Optimization: The system queries the Semantic Feature Bank to find the most similar features for each patch in the rendered view. The optimization objective minimizes the distance between the rendered view's features and the retrieved bank features.
Constraint Enforcement: This process enforces implicit structural constraints, compelling the 3DGS model to produce semantically coherent and artifact-free renders. The approach is model-agnostic, meaning it can be integrated with various 3DGS backbones (e.g., 3DGS, Compact 3DGS, Mip-Splatting) without modifying their core architectures.

Key Contributions

Semantic Feature Generation: A novel pipeline combining image-to-video diffusion models and Vision Foundation Models to generate high-quality semantic features. These features act as implicit structural cues for novel views in scenes lacking traditional texture and edge information.
Novel-View Semantic Learning: A learning strategy that allows 3DGS to optimize novel views without ground-truth supervision. By minimizing the distance between rendered features and a pre-constructed semantic feature bank, the method enforces structural consistency and reduces artifacts in glow regions.
NightGlow Dataset: The introduction of a new dataset specifically designed for evaluating 3D scene reconstruction in nighttime settings with strong glow effects. It comprises 18 scenes with approximately 540 images, addressing the lack of suitable benchmarks for this specific domain.

Experimental Results

The authors conducted extensive experiments on the new NightGlow dataset, as well as existing datasets containing glow effects (RawNeRF-Glow and Bilarf-Glow).

Quantitative Performance: Integrating GlowGS with existing 3DGS backbones yielded significant improvements. Specifically, when applied to Mip-Splatting (MGS), the method achieved an average PSNR of 28.24, a 1.78 dB improvement over the baseline MGS (26.46). Similar gains were observed for 3DGS (+1.69 dB) and Compact 3DGS (+1.56 dB).
Qualitative Performance: Visual comparisons demonstrate that GlowGS effectively reconstructs glow regions with smoother light diffusion, eliminating the duplicated light sources and unnatural bright spots common in baseline methods.
Ablation Studies:
- Diffusion Models: Using different image-to-video models (Pika vs. PromeAI) consistently improved performance over the baseline, validating the utility of the generated semantic features.
- VFMs: DINO was found to be the most effective VFM for feature extraction in glow scenes, outperforming CLIP and ViT, likely due to its robust feature representations learned from self-supervised data.
- Training Views: The method demonstrated robustness across varying numbers of training views (2 to 10), consistently outperforming baselines.
- Region Analysis: Separate evaluation of glow and non-glow regions confirmed that the method significantly improves metrics (PSNR, SSIM, LPIPS) specifically in the challenging glow areas.

Significance and Claims

The paper claims that GlowGS represents a significant advancement in 3D scene reconstruction for nighttime environments. By shifting the focus from relying solely on explicit geometric cues (which are absent in glow regions) to leveraging implicit semantic features, the method overcomes a fundamental limitation of current 3DGS approaches.

The authors emphasize that their approach is model-agnostic, making it a versatile enhancement for various 3DGS techniques. The introduction of the NightGlow dataset is highlighted as a crucial contribution to the field, providing a necessary benchmark for future research in nighttime reconstruction. The work demonstrates that combining generative models with foundation models can effectively compensate for missing structural information, enabling high-fidelity novel view synthesis in complex, low-texture nighttime scenes.