💻 computer science

CFSR: Geometry-Conditioned Shadow Removal via Physical Disentanglement

CFSR is a novel shadow removal framework that achieves state-of-the-art performance by reframing the task as a physics-constrained restoration process, which integrates 3D geometric cues and large-scale foundation model semantics to balance localized texture recovery with global illumination consistency.

Original authors: Pan Wang, Yihao Hu, Xiujin Liu, Hang Wang

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pan Wang, Yihao Hu, Xiujin Liu, Hang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a beautiful photograph of a park, but a large tree has cast a dark, messy shadow over a bench and a flower bed. Your goal is to "erase" that shadow and see what the bench and flowers really look like underneath.

This is the problem of Shadow Removal.

For a long time, computers tried to solve this like a simple "paint-by-numbers" task. They would look at the dark pixels and just try to make them lighter. But this often leads to mistakes: the computer might accidentally make the dark wood of the bench look like a dark shadow, or it might make the restored area look blurry and fake, like a plastic toy.

The paper you shared introduces a new method called CFSR. Think of CFSR not as a painter, but as a forensic detective who uses physics and 3D modeling to solve the crime of the missing light.

Here is how CFSR works, broken down into simple analogies:

1. The Problem: The "Black Box" vs. The "Physics Detective"

Old Way (The Black Box): Imagine a student trying to guess the answer to a math problem just by memorizing patterns. They might get lucky, but if the problem changes slightly, they fail. Old shadow-removal AI works like this: it guesses what the shadow-free image should look like based on what it has seen before. It doesn't understand why the shadow is there.
CFSR (The Physics Detective): CFSR knows that shadows aren't magic; they are caused by 3D objects blocking light. Instead of just guessing, CFSR asks: "What does the 3D shape of this object look like? How would light hit it?" It uses the laws of physics to reverse-engineer the shadow.

2. The Toolkit: Three Special Glasses

To solve the mystery, CFSR puts on three different pairs of "glasses" to see the world differently:

Glasses #1: The 3D X-Ray (Geometry Priors)
- What it does: It looks at the flat 2D photo and instantly figures out the 3D shape of everything (like a depth map). It knows where the ground is, where the wall is, and how the surface curves.
- The Analogy: Imagine looking at a flat drawing of a sphere. A normal person sees a circle. CFSR sees a sphere. It knows that light hits a sphere differently than a flat wall. This helps it avoid making the shadow look weird on curved surfaces.
Glasses #2: The "Big Picture" Brain (Semantic Priors)
- What it does: It uses a massive, pre-trained AI brain (called CLIP and DINO) that has "seen" millions of images. It knows what a "bench" or a "flower" looks like, even if the shadow is so dark you can't see the details.
- The Analogy: If you cover half a cat's face with a black cloth, you still know it's a cat. If the shadow covers a flower, CFSR's "Big Picture Brain" says, "I know this is a red rose, so I will hallucinate (guess) the red petals back in, even though the pixels are gone."
Glasses #3: The Noise Filter (HVI Color Space)
- What it does: Shadows are often very dark and "noisy" (grainy). CFSR changes the way it looks at colors. Instead of looking at Red, Green, and Blue, it looks at Hue, Value, and Intensity.
- The Analogy: Imagine trying to hear a whisper in a loud storm. CFSR puts on noise-canceling headphones that specifically filter out the "static" caused by the darkness, letting the true colors shine through.

3. The Secret Sauce: How It Puts It All Together

CFSR has two main tricks to make sure the final image looks real:

Trick A: The "Traffic Cop" (Dual Explicit Guided Attention)

When the AI tries to fix the image, it has to decide: "Should I copy the texture from the left side or the right side?"

Old AI: Might copy a brick texture onto a smooth wall because they look similar in color.
CFSR: The "Traffic Cop" checks the 3D shape and the object type. It says, "Wait! The 3D shape here is flat, but the texture on the left is for a brick wall. They don't match. Don't mix them!" It forces the AI to only blend pixels that make physical sense together.

Trick B: The "Two-Track Train" (Frequency Collaborative Reconstruction)

Restoring an image is hard because you need two things at once:

Smoothness: The overall lighting needs to be soft and even (Low Frequency).
Sharpness: The edges of objects need to be crisp and clear (High Frequency).

Old AI: Tries to do both at once, often resulting in a blurry mess or a jagged, noisy image.
CFSR: Splits the work into two trains.
- Train 1 (Smooth): Fixes the big, soft lighting issues.
- Train 2 (Sharp): Fixes the tiny, sharp edges and textures.
- The Merge: It combines them perfectly, ensuring the lighting is smooth and the edges are sharp, without one ruining the other.

The Result

When you use CFSR, the result isn't just a "lightened" photo. It's a physically correct photo.

The shadows disappear naturally.
The colors don't look washed out or weird.
The edges of objects stay sharp.
If a shadow was hiding a detail (like a logo on a box), CFSR can "hallucinate" it back in correctly because it understands what the object is.

In summary: CFSR stops treating shadow removal like a simple photo filter. Instead, it treats it like a 3D physics simulation, using the shape of the world and the "knowledge" of a giant AI brain to reconstruct the scene exactly as it would look if the sun were shining directly on it.

1. Problem Statement

Shadow removal is a critical pre-processing step for high-level computer vision tasks (e.g., object detection, autonomous driving). However, existing deep learning approaches largely treat this as a deterministic "image-to-image" mapping problem within the 2D RGB domain. This "black-box" paradigm suffers from several limitations:

Lack of Physical Interpretability: Models struggle to distinguish between intrinsic dark textures (albedo) and actual shadows, often failing to balance local texture recovery with global illumination consistency.
2D-3D Domain Gap: Current methods ignore the underlying 3D physical geometry that drives shadow formation, leading to artifacts in complex scenes.
Frequency Domain Issues: Without physics-constrained inversion, methods often compromise global lighting for local details, resulting in spectral discrepancies and visual artifacts.

The authors argue that shadow removal should be reframed from blind pixel regression to a physics-constrained degradation inversion problem, where the restored image must adhere to the physical manifold of a latent clean scene.

2. Methodology: CFSR Framework

The proposed CFSR (Geometry-Conditioned Shadow Removal) is a multi-modal prior-driven framework that integrates 3D geometric cues with large-scale foundation model semantics. The architecture consists of four main stages:

A. 3D Prior Estimation & Semantic Extraction

Geometric Priors: A pre-trained monocular depth estimator generates a pseudo-depth map ( $D$ ). Using a normalized intrinsic matrix, this is converted into a 3D point cloud ( $P$ ) and surface normal map ( $N$ ). These provide explicit structural constraints.
Semantic Priors: A frozen DINOv2 encoder extracts dense, region-level semantic features ( $F_{dino}$ ) that are robust to illumination changes.

B. Multi-Modal Feature Initialization (HVI Space)

To decouple illumination from chrominance and suppress noise in deep shadows:

Custom HVI Color Space: The input RGB is transformed into a polar representation ( $H_x, H_y, V$ ).
Intensity Collapse Mechanism (ICM): A learnable power-law transformation is applied to the Value channel ( $V$ ). This dynamically suppresses sensor noise in low-light (shadowed) regions where $V \to 0$ , preventing the amplification of noise during feature extraction.
Fusion: The noise-suppressed HVI tensor is concatenated with 3D geometric priors ( $P, N$ ) to form the initial multi-modal embedding.

C. Physics-Guided Feature Extraction

The core innovation is the Geometric & Semantic Dual Explicit Guided Attention (GSGA) mechanism:

Explicit Guidance: Unlike standard self-attention, GSGA modulates the attention affinity matrix using two explicit guidance matrices:
1. Geometric Guidance ( $G_{geo}$ ): Computed via cosine similarity of 3D surface normals within local windows. It enforces that features are aggregated only from co-planar regions, preventing bleeding across occlusion boundaries.
2. Semantic Guidance ( $G_{sem}$ ): Computed via similarity of DINO features. It preserves texture fidelity by ensuring semantically consistent regions are attended to, even if they share a plane.
Global Semantic Injection: At the network bottleneck, a frozen CLIP visual encoder injects holistic scene semantics via Cross-Modal Attention. This helps "hallucinate" and recover textures in severely degraded shadow regions where local information is lost.

D. Frequency Collaborative Reconstruction Module (FCRM)

To resolve the trade-off between sharp boundaries and smooth illumination, the decoding process is disentangled:

High-Frequency Branch: Uses Dense CNNs to reconstruct sharp structural details and occlusion boundaries.
Low-Frequency Branch: Uses a Mini Transformer U-Net with Haar Wavelet Transforms (DWT) to model global illumination consistency and smooth variations.
Collaborative Fusion: The branches are fused under continuous 3D geometric conditioning (using surface normals) to ensure the final output maintains both photorealistic texture and physical lighting consistency.

3. Key Contributions

Physics-Aware Formulation: Shifts the paradigm from unconstrained 2D mapping to a physics-constrained restoration process, utilizing a custom HVI space with an Intensity Collapse mechanism to robustly fuse RGB data with 3D priors.
Geometric & Semantic Dual Explicit Guided Attention (GSGA): A novel attention mechanism that structurally enforces physical lighting constraints by modulating the affinity matrix with 3D surface normals and DINO semantic maps, effectively bridging the 2D-3D gap.
Geometry-Conditioned Collaborative Decoding: The FCRM disentangles high-frequency texture recovery from low-frequency illumination smoothing, guided by 3D geometry and global CLIP semantics, achieving an optimal balance between detail and consistency.

4. Experimental Results

CFSR was evaluated on six challenging benchmarks: ISTD, ISTD+, SRD, INS, WSRD+, and Ambient6K.

Quantitative Performance: CFSR achieves State-of-the-Art (SOTA) performance across almost all metrics.
- On ISTD, it achieves 30.91 dB PSNR and 0.979 SSIM.
- On the complex Ambient6K dataset (multi-source indirect lighting), it significantly outperforms existing methods with 23.62 dB PSNR and 0.843 SSIM.
Generalization: In cross-dataset evaluations (e.g., training on ISTD+ and testing on SRD), CFSR demonstrates superior transferability, outperforming competitors like OmniSR by 0.33 dB in specific transfers. This confirms the model learns physically meaningful priors rather than dataset-specific biases.
Qualitative Analysis: Visual comparisons show CFSR effectively eliminates boundary artifacts and residual shadows while preserving original chromaticity and texture, outperforming generative methods that often cause color degradation or over-smoothing.

5. Significance

This work represents a significant shift in low-level vision tasks by integrating foundation model priors (DINO, CLIP) with explicit physical constraints (3D geometry).

Theoretical Impact: It demonstrates that reframing restoration as a physics-constrained inversion problem yields more robust and interpretable results than deterministic mapping.
Practical Impact: The ability to handle complex, multi-source lighting and generalize across unseen datasets makes CFSR highly suitable for real-world applications like autonomous driving and robotics, where lighting conditions are unpredictable.
Methodological Innovation: The introduction of the GSGA and FCRM modules provides a new blueprint for how to structurally embed physical laws into deep learning architectures, potentially applicable to other inverse problems in computer vision.