💻 computer science

WildShadowRemover: In-the-Wild Video Shadow Removal via Detail-Preserving Video Diffusion Models

The paper introduces WildShadowRemover, a framework that leverages LoRA-fine-tuned video diffusion models augmented with detail-injection and frequency-decomposed modulation modules alongside depth priors to achieve robust, high-quality video shadow removal in complex real-world scenarios, supported by the newly constructed large-scale WildShadow dataset.

Original authors: Jiamin Xu, Cong Wang, Zheng Dong, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiamin Xu, Cong Wang, Zheng Dong, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a beautiful, sunny day through a window, but someone has spilled a bucket of dark, muddy water all over the glass. The world outside is still there, vibrant and full of life, but the mud distorts everything, making it hard to see the details or enjoy the view. In the world of computer vision—the science of teaching computers to "see" and understand images—shadows are a lot like that spilled mud. They are a natural part of how light works, but they can mess up important tasks like helping a self-driving car spot a pedestrian or helping a video editor clean up a scene. For a long time, computers have been pretty good at cleaning up shadows in single, still photos, like scrubbing one muddy window pane. But when it comes to moving pictures, or videos, it's a whole different ballgame. A video is like a stream of hundreds of window panes flowing one after another; if you clean them one by one without thinking about the flow, the result looks jittery and broken, like a flickering lightbulb. The big challenge is figuring out how to wash away the shadow mud from an entire video stream without smearing the picture or making the scene look like a glitchy cartoon.

This is where a new tool called WildShadowRemover steps in, acting like a super-smart, time-traveling janitor for video. The researchers behind this project realized that to clean up shadows in the wild (meaning messy, real-world scenes like busy streets or forests), you need more than just a scrub brush; you need a deep understanding of how objects move and how light behaves over time. They built their system on top of a "video diffusion model," which is a type of artificial intelligence that has already learned how to create videos from scratch by studying millions of hours of footage. Think of this AI as a master painter who has seen every kind of shadow and every kind of object in the world. Instead of teaching the AI to paint from scratch, the researchers used a clever trick called "LoRA fine-tuning." Imagine this as giving the master painter a specialized, lightweight apron that teaches them specifically how to remove shadows without forgetting how to paint the rest of the picture.

However, the researchers noticed that while this AI painter was great at removing the big dark blobs of shadow, it sometimes forgot the tiny details, like the texture of a brick wall or the pattern on a shirt, making the video look a bit blurry. To fix this, they added a "detail injection module." You can think of this as a high-tech magnifying glass that looks at the original shadowed video, picks out the sharp, crisp details, and carefully pastes them back onto the cleaned-up version. But there's a catch: if you just paste the old details back, you might accidentally paste the shadow back in too! So, the team invented a "shadow-mask-guided frequency-decomposed modulation" system. That's a mouthful, but imagine it as a smart filter that sorts the image into two piles: the "smooth, low-frequency" parts (like the general shape of a tree) and the "crunchy, high-frequency" parts (like the leaves). The system then uses a map of where the shadows are to say, "Keep the crunchy leaves, but throw away the crunchy shadows."

To make sure the video looks right even when the lighting is weird or the camera is moving, the system also uses a "depth map" from a tool called Depth Anything 3. This is like giving the AI a 3D ruler to understand how far away things are, so it knows that a shadow on the ground is different from a shadow on a wall. The researchers didn't just build the tool; they also built a massive training ground for it called WildShadow. Since real-world videos with perfect "before and after" pairs are hard to find, they created a giant library of 6,100 synthetic video clips (4,500 for training and 600 for testing) using 3D computer graphics. These clips cover everything from indoor rooms to urban streets and natural landscapes, ensuring the AI learns to handle all kinds of messy, real-world scenarios.

The results suggest that this new method is quite effective. When tested, WildShadowRemover didn't just remove the shadows; it kept the video smooth and consistent, so the cleaned-up scenes didn't flicker or jump around. It managed to preserve the fine details that other methods often lose, producing videos that look natural and clear. The authors found that by combining the powerful "imagination" of the pre-trained video AI with these specific, detail-focused tools, they could handle complex, unpredictable lighting conditions much better than previous approaches. While the paper focuses on synthetic data for training and testing, the results indicate that this approach could significantly improve how computers handle shadows in the real world, making videos cleaner and more useful for everything from entertainment to safety systems.

Technical Summary: WildShadowRemover

Problem Statement
Video shadow removal in unconstrained, "in-the-wild" environments remains a significant challenge due to complex illumination, diverse shadow appearances, and a scarcity of high-quality training data. While shadows provide valuable cues for scene understanding, they adversely affect applications such as object tracking, intrinsic decomposition, and image editing. Existing methods face two primary limitations:

Data Scarcity: Paired shadow/shadow-free datasets are limited in scale and scene diversity. Early datasets (e.g., ISTD, SRD) rely on controlled conditions, while recent synthetic datasets (e.g., OmniSR, Infinigen) focus primarily on indoor environments, failing to cover the complexity of urban streets and UAV-captured scenes.
Model Limitations: Current approaches often rely on single-image diffusion models or task-specific architectures that struggle to model temporal consistency. While image diffusion models have shown promise, they cannot explicitly capture the spatiotemporal correlations between objects and their shadows across video frames.

Methodology
The authors propose WildShadowRemover, a framework that adapts a pretrained video diffusion model (specifically the Wan-Control variant) for robust video shadow removal. The method operates in the latent space and employs a two-stage strategy to balance generative power with detail preservation:

Latent Video Diffusion with LoRA Fine-Tuning:
- The framework utilizes a pretrained Wan-Control model, which is based on a flow-matching paradigm and supports controllable video generation using RGB and depth conditions.
- Instead of training from scratch, the authors employ LoRA (Low-Rank Adaptation) to fine-tune the Diffusion Transformer (DiT) blocks. This allows for parameter-efficient adaptation while keeping the majority of the pretrained weights frozen.
- The model is conditioned on the input shadow video ( $X$ ) and monocular depth maps ( $D$ ) estimated by Depth Anything 3 (DA3). This geometry-aware guidance helps the model understand scene structure under challenging lighting.
- The diffusion process predicts a coarse, shadow-free latent representation, which is decoded into a coarse shadow-free video ( $\hat{Y}_0$ ).
Detail Injection and Frequency-Decomposed Modulation:
- To recover fine-grained textures lost during the diffusion process, the authors introduce a Detail Injection Module that augments the frozen VAE decoder.
- To prevent the reintroduction of shadow artifacts during detail injection, they propose a Shadow-Mask-Guided Frequency-Decomposed Modulation (FDM) module.
- Mechanism: A soft shadow mask is estimated by comparing the input shadow video with the coarse prediction. The FDM module decomposes features into low-frequency and high-frequency components. It then selectively modulates these components using affine parameters predicted from the shadow mask. This allows the model to restore high-frequency textures (details) while suppressing high-frequency shadow artifacts.
- Geometry priors from DA3 are also integrated into this module to guide detail restoration under complex illumination.

Key Contributions

WildShadowRemover Framework: A diffusion-based framework for in-the-wild video shadow removal that combines LoRA fine-tuning of a pretrained video foundation model with a novel detail injection mechanism. It effectively preserves fine textures while removing complex shadows and maintaining temporal consistency.
WildShadow Dataset: The construction of WildShadow, a large-scale synthetic paired video shadow removal dataset. It covers diverse indoor (3D-FRONT, HSSD), urban outdoor (iCity), and natural outdoor (Infinigen) environments.
Benchmark: The introduction of a comprehensive benchmark for evaluating generalization across diverse synthetic scenes, including HSSD and newly constructed iCity scenes.
Performance: Extensive experiments demonstrate that the method achieves state-of-the-art performance on both synthetic and real-world benchmarks, significantly outperforming existing approaches in shadow removal quality, temporal consistency, and generalization.

Results and Claims
The paper reports that WildShadowRemover produces temporally coherent, shadow-free videos with superior visual quality compared to existing methods. Key findings include:

Temporal Consistency: By leveraging the spatiotemporal priors of the pretrained video diffusion model, the method maintains coherent scene content and shadow evolution consistent with object motion, a capability lacking in single-image methods.
Detail Preservation: The frequency-decomposed modulation module successfully restores high-frequency details (e.g., thin structures, textures) without reintroducing shadow patterns.
Generalization: The method demonstrates strong generalization across challenging in-the-wild scenarios, including urban streets and UAV-captured videos, attributed to the diversity of the WildShadow dataset and the robustness of the pretrained foundation model.
Versatility: Although designed for video, the approach can be directly applied to single-image shadow removal tasks without architectural modification, showing competitive results on standard image benchmarks (ISTD+, SRD, INS).

Significance
The authors position this work as a step toward solving the "in-the-wild" video shadow removal problem, which has been largely unexplored due to data and model limitations. By adapting a large-scale video foundation model and introducing geometry-aware, frequency-selective detail restoration, the framework addresses the critical need for temporally consistent and high-fidelity shadow removal. This capability is highlighted as beneficial for downstream applications such as 3D content creation and intrinsic decomposition, where consistent shadow-free video is essential. The paper emphasizes that their approach moves beyond task-specific architectures to leverage the intrinsic correlations between objects and shadows learned by large-scale video pretraining.

Technical Summary: WildShadowRemover

More like this