Unifying Color and Lightness Correction with View-Adaptive Curve Adjustment for Robust 3D Novel View Synthesis

Luminance-GS++ is a novel 3D Gaussian Splatting framework that achieves robust 3D novel view synthesis under diverse illumination conditions by integrating view-adaptive lightness adjustment with local residual refinement, all while preserving the explicit 3DGS formulation for real-time rendering.

The Gist

The Big Problem: The "Bad Photo" Group Project

Imagine you and your friends are trying to build a perfect 3D model of a park using photos taken from different angles. This is what computers do in Novel View Synthesis (NVS)—they take 2D pictures and stitch them together to create a 3D world you can walk through.

However, in the real world, taking these photos is messy:

• The Lighting is Weird: One photo was taken at noon (too bright), another at dusk (too dark), and another under a streetlamp (too orange).
• The Cameras are Different: One friend's phone has a "cool" blue tint, while another's has a "warm" yellow tint.
• The Result: When the computer tries to stitch these mismatched photos together, the 3D model looks broken. You get "ghosts" (floating blurry shapes), weird color shifts, and a scene that doesn't look real.

Existing methods try to fix this, but they are either too slow (like trying to paint a masterpiece pixel-by-pixel) or they break the 3D structure when they try to fix the colors.

The Solution: Luminance-GS++ (The "Smart Photo Editor")

The authors propose a new system called Luminance-GS++. Think of it as a super-smart photo editor that works while the 3D model is being built, rather than trying to fix the photos before or after.

Here is how it works, broken down into three simple steps:

1. The "Global Adjuster" (The Master Dimmer Switch)

First, the system looks at the whole group of photos. It realizes, "Hey, this photo is too dark, and that one is too blue."

• The Analogy: Imagine a master dimmer switch and a color filter for the whole room. The system applies a global curve to brighten up the dark photos and a color matrix to neutralize the weird tints.
• What it does: It makes sure all the photos are roughly on the same "level" of brightness and color, so the computer can start comparing them fairly.

2. The "Local Refiner" (The Pixel-Level Surgeon)

The global adjuster is good, but it's a bit blunt. It treats every pixel in a dark area the same way. But what if one part of the photo is a shiny reflection (specular highlight) that needs a different fix than the shadow next to it?

• The Analogy: This is where the Local Refinement comes in. Think of it as a surgeon with a tiny scalpel. After the global dimmer switch is flipped, this module looks at the image pixel-by-pixel to fix tiny details. It adds a "residual map" (a layer of tiny corrections) to fix specific spots that the global switch missed, like a smudge on a lens or a weird glare.
• Why it matters: This prevents the "ghosts" and floating artifacts that happen when the computer gets confused by tiny lighting differences.

3. The "Teamwork" (Joint Optimization)

The coolest part is that Luminance-GS++ doesn't just fix the photos and then build the 3D model. It does both at the same time.

• The Analogy: Imagine a construction crew building a house while a painter is simultaneously fixing the walls. If the painter notices a wall is crooked, they tell the builders, and the builders adjust the bricks. If the builders move a wall, the painter adjusts their brushstrokes.
• The Result: The 3D model and the color corrections learn from each other. The model gets more accurate geometry, and the colors get more consistent, all without slowing down the process.

Why Is This a Big Deal?

1. It's Fast: Old methods (like NeRF) are like painting a photo with a single hairbrush—slow and detailed. This new method uses 3D Gaussian Splatting, which is like throwing thousands of tiny, colorful confetti pieces to form the image instantly. Luminance-GS++ keeps this speed while adding the ability to fix bad lighting.
2. It's Robust: It works even if you take photos in a cave (pitch black), under the sun (blindingly bright), or with mixed lighting (half in shadow, half in sun).
3. It's Consistent: It ensures that if you walk around the 3D model, the colors don't suddenly shift from blue to orange. It keeps the "mood" of the scene consistent.

The Bottom Line

Luminance-GS++ is like giving a 3D construction crew a team of expert photographers and editors who work in real-time. They don't just build the 3D world; they fix the lighting and color issues as they build, ensuring the final result looks like a high-quality, realistic movie scene, even if the original photos were taken in terrible conditions.

It solves the problem of "garbage in, garbage out" by turning "garbage photos" into "gold-standard 3D models" without sacrificing speed.

Technical Summary

1. Problem Statement

Modern 3D Novel View Synthesis (NVS) methods, particularly Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), rely heavily on the assumption of photometric consistency across multi-view inputs. However, in real-world capture scenarios, this assumption is frequently violated due to:

• Illumination Variations: Differences in light intensity (low-light, overexposure) and color temperature (white balance shifts).
• Sensor & ISP Limitations: Variations in camera sensors and Image Signal Processor (ISP) configurations introduce systematic chromatic and luminance inconsistencies.

These inconsistencies lead to severe reconstruction artifacts, including floaters, ghosting, geometric instability, and color shifts. Existing solutions face two main limitations:

1. Preprocessing approaches: Applying 2D image enhancement independently to each view disrupts radiometric consistency, causing 3D reconstruction failures.
2. Implicit representation methods (NeRF-based): While some NeRF variants disentangle appearance from geometry, they are often constrained by training data lighting conditions, lack generalization to novel illumination scenarios, and suffer from slow training/rendering speeds compared to explicit methods.

2. Methodology: Luminance-GS++

The authors propose Luminance-GS++, a framework built upon 3D Gaussian Splatting (3DGS) that performs robust NVS under diverse degradation conditions without modifying the explicit 3DGS representation. Instead, it treats illumination correction as a pseudo-label optimization problem.

Core Architecture

The method jointly optimizes the 3D Gaussian attributes and a pseudo-enhancement pipeline that transforms degraded inputs into consistent, well-lit pseudo-images. The pipeline consists of two stages:

1. Global Adjustment (View-Adaptive):

   • Per-View Color Matrix ($M_k$): A learnable $3\times3$ matrix applied to each view to correct global color shifts (white balance).
   • View-Adaptive Curve Adjustment: A shared global tone curve ($L_g$) combined with a view-specific bias ($L^b_k$) generated by an attention module. This normalizes brightness and exposure across views while allowing for view-specific nuances.
   • Output: A globally adjusted image that fixes major exposure and color temperature issues.

2. Local Refinement (Pixel-Wise Residual):

   • To address residual artifacts and spatially varying inconsistencies that global curves miss, a Residual Branch ($R$) is introduced.
   • This branch uses three ConvNeXt blocks to predict a pixel-wise residual map ($C^{res}_k$).
   • The final pseudo-enhanced image is the sum of the globally adjusted image and the residual map: $C^{out}_k = \text{GlobalAdjust}(C^{in}_k) + R(C^{in}_k)$.

Training Strategy & Loss Functions

The framework employs unsupervised objectives to jointly optimize the 3D Gaussians and the enhancement modules:

• Dual Rendering: The model renders both the original input view ($\hat{C}^{in}$) and the pseudo-enhanced view ($\hat{C}^{out}$) from the same 3D Gaussians.
• Reconstruction Loss ($L_{reg}$): Minimizes the difference between rendered images and their respective targets (input and pseudo-enhanced).
• Spatial Consistency Loss ($L_{spa}$): Inspired by Zero-DCE, this enforces that the relative intensity relationships between neighboring pixels in the enhanced image match those in the input, preserving structural details.
• Color Correction Loss ($L_{cc}$): Based on the "Shade-of-Gray" algorithm, this loss minimizes color discrepancies across views to enforce cross-view color constancy.
• Curve Regularization ($L_{curve}$): Constrains the learned tone curves to follow plausible physical priors (power-law and S-curves) and ensures smoothness via Total Variation (TV) loss.

3. Key Contributions

1. Luminance-GS++ Framework: A novel 3DGS-based method that unifies global view-adaptive curve adjustment with local pixel-wise residual refinement. It improves robustness to lightness and color degradations while preserving the real-time rendering efficiency of standard 3DGS.
2. Unsupervised Multi-View Consistency: The design of specific loss functions ($L_{spa}, L_{cc}$) that enforce radiometric and geometric consistency across views during the enhancement process, preventing the "floaters" and artifacts common in independent preprocessing.
3. State-of-the-Art Performance: The method achieves superior results across challenging scenarios (low-light, overexposure, mixed color/illumination variations) without requiring RAW data or modifying the underlying 3D Gaussian representation.

4. Experimental Results

The authors evaluated Luminance-GS++ on the LOM dataset (real-world low-light/overexposure) and Mip-NeRF 360 (synthetic variations).

• Quantitative Performance:

  • Low-Light (LOM): Luminance-GS++ achieved a mean PSNR of 20.74 dB, significantly outperforming the previous best (Luminance-GS at 18.34 dB) and NeRF-based methods like Aleth-NeRF (19.87 dB).
  • Overexposure (LOM): Achieved a mean PSNR of 21.41 dB, surpassing all baselines.
  • Color Degradation: Demonstrated superior chromaticity consistency, producing compact color clusters in CIELAB space compared to the dispersed distributions of baselines.
  • Mixed Degradation: Consistently outperformed all competitors in combined lightness and color variation scenarios.

• Efficiency:

  • Training: Luminance-GS++ trains in 0.23 GPU hours (LOM) and 0.28 GPU hours (Mip-NeRF 360), which is orders of magnitude faster than NeRF-based methods (e.g., Aleth-NeRF takes ~14 hours).
  • Rendering: Maintains real-time rendering speeds comparable to vanilla 3DGS, unlike NeRF-based approaches which are significantly slower.

• Ablation Studies:

  • Removing the Residual Branch caused performance to collapse in complex mixed-degradation scenarios (PSNR dropped from 19.56 to 15.96), proving its necessity for fine-grained detail.
  • Removing Spatial Consistency Loss led to significant quality degradation, highlighting its role in preserving structure.

5. Significance

This work bridges the gap between computational photography (image enhancement) and 3D reconstruction. By integrating illumination correction directly into the 3DGS training loop via a pseudo-label strategy, Luminance-GS++ solves the "photometric inconsistency" bottleneck that plagues real-world NVS.

Its significance lies in:

• Practicality: It enables high-quality 3D reconstruction from consumer-grade cameras in uncontrolled lighting (e.g., low-light or mixed-color environments) without needing specialized hardware or RAW data.
• Efficiency: It retains the speed advantages of 3DGS, making it viable for real-time applications like AR/VR and autonomous driving where lighting conditions are unpredictable.
• Generalizability: The unsupervised nature of the approach allows it to generalize to novel illumination scenarios that were not present in the training set.