💻 computer science

SRUG: Shadow-Guided Relightable Urban Scene with Generation Model

This paper presents SRUG, a novel framework that addresses the challenges of relighting unbounded urban scenes by leveraging shadows to guide 3D geometry completion and employing an iterative large material model for robust material decomposition, ultimately enabling physically reasonable relighting and novel view synthesis.

Original authors: Yonghao Zhao, Zexin Yin, Jian Yang, Beibei Wang, Jin Xie

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yonghao Zhao, Zexin Yin, Jian Yang, Beibei Wang, Jin Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photo album of a busy city street. You want to create a 3D movie from these photos where you can change the time of day, move the sun, or swap a sunny sky for a stormy one. This is called "relighting."

However, building a city in 3D is like trying to build a model of a forest while only looking at a few trees. You can't see the trees behind the ones in front of you, but those hidden trees still cast shadows on the ones you can see. If you don't account for the invisible trees, your shadows will look wrong, and your 3D city will look fake.

The paper introduces SRUG, a new system designed to solve this specific problem. Think of SRUG as a "smart city builder" that uses three main tricks to make the city look real under any lighting condition.

1. The "Shadow Detective" (Shadow-Guided Completion)

The Problem: In a real city, buildings and cars block your view. The computer can't see the "invisible" parts of the street, but those invisible parts cast shadows on the visible parts. If the computer doesn't know what's hiding behind the wall, it can't calculate the shadow correctly.

The SRUG Solution: SRUG acts like a detective who uses shadows as clues.

The Metaphor: Imagine you are in a dark room and you see a shadow on the wall. Even if you can't see the object casting it, the shape of the shadow tells you exactly what the object looks like.
How it works: SRUG uses a special "3D completion model" (a type of AI that guesses missing 3D shapes) to fill in the invisible parts of the city. But instead of just guessing randomly, it uses the shadows as a guide. It asks, "If this shadow exists, what invisible building must be there to cast it?" This ensures that the hidden parts of the city are built in a way that makes physical sense, so the shadows look real when the sun moves.

2. The "Iterative Art Critic" (Iterative Material Decomposition)

The Problem: To change the lighting, the computer needs to separate the "paint" of the city (the material, like red brick or shiny glass) from the "light" hitting it.

The Issue: If you just look at a photo of a red brick wall in the sun, the computer might think the wall is actually glowing orange because of the sun, rather than just being red brick. This is called "lighting bake-in."
The SRUG Solution: SRUG uses a "Large Material Model" (LMM), which is like a super-smart art critic trained on millions of images to know what materials look like.
The Metaphor: Imagine the art critic is trying to guess the color of a wall, but the wall is covered in a weird filter.
- Step 1: The critic makes a guess based on the filtered photo.
- Step 2: The computer builds a 3D model based on that guess.
- Step 3: The computer shows the critic the new 3D model.
- Step 4: The critic says, "Ah, now that I see the 3D shape, I realize my first guess was a bit off. Let me refine my guess."
- Repeat: They do this back-and-forth a few times. This "iterative" process helps the computer bridge the gap between the AI's training (which was mostly on fake, computer-generated images) and the messy reality of a real city street. The result is a clean separation of the wall's true color from the sunlight.

3. The "Physics Engine" (Physically-Based Lighting)

The Problem: Real cities have complex light: direct sunlight, soft light from the sky, and light bouncing off other buildings.
The SRUG Solution: SRUG doesn't just guess the color; it uses a physics-based formula. It calculates three specific types of light:

Sunlight: The direct, harsh beam from the sun.
Skylight: The soft, blue-ish light coming from the sky (which gets blocked by buildings, creating "ambient occlusion").
Indirect Light: Light that bounces off other surfaces.

By calculating these separately, SRUG can move the sun to a new position, and the shadows will shift realistically, and the light bouncing off the buildings will update automatically.

The Result

When you put these three tricks together, SRUG can take a set of photos or a video of a city and turn it into a 3D world where:

Shadows are real: Even the shadows cast by things you can't see in the original photos are calculated correctly.
Materials are accurate: A brick wall stays red, even if you change the sun to a sunset.
No "floating" artifacts: The 3D model doesn't have weird floating bits of dust or geometry because the shadow guidance keeps the geometry grounded.

In short, SRUG is a tool that uses shadows to "see" the invisible parts of a city and uses a smart, repeating feedback loop to figure out exactly what the city is made of, allowing you to change the weather and time of day in a way that looks physically real.

Technical Summary: SRUG: Shadow-Guided Relightable Urban Scene with Generation Model

Problem Statement

Reconstructing relightable 3D urban scenes from multi-view images or videos is a critical task for applications such as autonomous driving, digital twins, and world modeling. However, this task is inherently ill-posed due to the difficulty of separating lighting from material properties. Urban environments present specific challenges that exacerbate these issues:

Unbounded Nature: Urban scenes extend beyond visible regions. Invisible geometry can cast shadows onto visible areas, making accurate shadow modeling difficult without recovering these hidden regions.
Complex Illumination: Urban lighting involves a mix of sunlight, skylight, and indirect light, leading to severe ambiguities in material decomposition.
Sparse Views: Limited input views further complicate the separation of materials and lighting, often resulting in "lighting bake-in" artifacts where lighting information is incorrectly encoded into the estimated albedo.
Limitations of Existing Methods: Previous approaches based on Neural Radiance Fields (NeRF) suffer from low expressiveness and high computational costs. Recent 3D Gaussian Splatting (3DGS) methods improve efficiency but often neglect invisible regions or struggle with robust material decomposition under complex urban illumination. Methods utilizing Large Material Models (LMM) often fail to bridge the domain gap between synthetic training data and real-world urban scenarios.

Methodology

The authors propose SRUG (Shadow-Guided Relightable Urban Scene with Generation Model), a framework designed to address these challenges through three core components:

1. Shadow-Guided Invisible Geometry Completion

To handle the unbounded nature of urban scenes, SRUG recovers the geometry of invisible regions using shadows as physical guidance.

3D Completion: The framework employs a pretrained 3D completion model (Difix3D) to generate pseudo-supervision for invisible regions by rendering novel views and repairing them based on reference images.
Differentiable Gaussian Shadow Mapping (DGSM): To ensure physical consistency, the authors introduce DGSM. Unlike standard shadow mapping which uses hard binary depth comparisons (breaking differentiability), DGSM reformulates visibility as a continuous, differentiable probability function: $V = \text{Sigmoid}(k(z_s - z))$ . This allows shadows to serve as supervisory signals for optimizing the 3D geometry of invisible areas, enabling the synthesis of physically plausible shadows.

2. Iterative Material Decomposition

To achieve robust material decomposition under complex lighting and sparse views, SRUG utilizes a Large Material Model (LMM) but addresses the synthetic-to-real domain gap.

LMM Supervision: A diffusion-based LMM (WeatherDiffusion) predicts material properties (albedo, metallic, roughness) from RGB images.
Iterative Refinement: Direct application of LMMs to real scenes often yields unrealistic results. SRUG proposes an iterative scheme where the LMM is conditioned on the current Gaussian material estimates. In each cycle, the LMM performs diffusion-based inpainting on the rasterized material maps from the Gaussians to refine the predictions. These refined maps then supervise the next optimization cycle. This process progressively bridges the domain gap, ensuring consistent and robust material decomposition.

3. Physically-Based Lighting and Shading

SRUG employs a physically-based lighting model to support reliable relighting.

Lighting Decomposition: Outgoing radiance is decomposed into three components: direct sunlight ( $L_{sun}$ ), skylight ( $L_{sky}$ ), and indirect light ( $L_{ind}$ ).
Deferred Shading: To avoid normal blurring artifacts common in forward shading, the framework uses deferred shading, rasterizing normals and material properties into G-buffers before shading.
Learnable Ambient Occlusion (AO): A learnable ambient occlusion attribute is introduced for each Gaussian primitive to approximate skylight occlusion caused by complex geometry.
Indirect Light Network: A neural network predicts indirect outgoing radiance based on G-buffers and direct lighting components.

Optimization Strategy

The framework is trained in two stages:

Initialization: Focuses on reconstructing visible geometry and appearance using standard 3DGS losses (color, normal consistency, regularization).
Relighting: Introduces material properties, light parameters, and the specific losses for shadow guidance, material consistency (via LMM), and visibility.

Key Contributions

The paper identifies three primary contributions:

SRUG Framework: A novel framework for constructing relightable urban scenes from multi-view images or videos that effectively disentangles lighting from material properties.
Shadow-Guided Invisible Geometry Completion: A module that leverages shadows as supervisory signals via DGSM to guide a 3D completion model in recovering invisible regions and synthesizing physically plausible shadows.
Iterative Material Decomposition: A scheme that utilizes an LMM to provide material supervision while iteratively refining material properties to bridge the domain gap between synthetic data and real-world urban scenes.

Results

The authors evaluated SRUG on both real-world datasets (KITTI-360, Waymo, Tanks and Temple) and a synthetic urban dataset.

Quantitative Performance: SRUG achieves state-of-the-art performance in Novel View Synthesis (NVS), relighting, and material decomposition across most metrics (PSNR, SSIM, LPIPS, MAE). Notably, it demonstrates superior robustness under sparse-view conditions (e.g., the TandT50 subset), outperforming competitors like UrbanIR, GS-IR, and R3DG.
Visual Quality: Visual comparisons show that SRUG effectively mitigates lighting bake-in artifacts in albedo estimation and produces consistent shadows under changing sun directions. It avoids floating artifacts and generates more reliable relighting results under environment map-based lighting compared to existing methods.
Efficiency: While the incorporation of 3D completion and iterative decomposition increases training time compared to purely Gaussian-based methods, SRUG achieves a 2× speedup in training and approximately 10× speedup in inference compared to UrbanIR.

Significance and Claims

The paper claims that SRUG significantly advances the state-of-the-art in urban scene relighting by addressing the specific challenges of unbounded environments and complex illumination.

Physical Consistency: By using shadows to guide geometry completion, the method ensures that invisible regions are reconstructed in a way that is physically consistent with the visible scene, a capability lacking in previous approaches.
Robust Decomposition: The iterative material decomposition scheme successfully overcomes the limitations of applying synthetic-trained LMMs to real-world data, providing a robust solution for separating materials from lighting without requiring additional sensors like LiDAR.
Practicality: The framework supports physically reasonable relighting under both directional sunlight and environment maps, making it suitable for applications requiring world models and digital twins.

The authors acknowledge limitations, noting that the iterative scheme may introduce multi-view inconsistencies in long-sequence inputs, leading to blurred material estimates, and that the current implementation does not natively handle dynamic scenes. Future work is suggested to address these via long-sequence LMMs and dynamic Gaussian representations.