PhaSR: Generalized Image Shadow Removal with Physically Aligned Priors

1. Problem Statement

Shadow removal is a fundamental computer vision task critical for downstream applications like segmentation, tracking, and 3D reconstruction. However, existing methods face three primary challenges:

• Ambiguity: Distinguishing between shadows (illumination changes) and intrinsic object darkness (material properties) is difficult, often leading to color distortion near textured boundaries.
• Generalization Gap: Most state-of-the-art (SoTA) methods perform well on single-light, direct shadow benchmarks but fail in complex real-world scenarios involving multi-source ambient lighting, diffuse indirect illumination, and color shifts.
• Prior Misalignment: Traditional encoder-decoder frameworks often lose physical priors through bottlenecks. Furthermore, existing methods that integrate geometric (depth/normal) and semantic features often fail to align them correctly. Geometric features are sensitive to local shading variations, while semantic features are stable across lighting; without proper alignment, geometric noise disrupts semantic consistency, or semantic smoothing erases illumination boundaries.

2. Methodology: PhaSR

The authors propose PhaSR (Physically Aligned Shadow Removal), a framework designed to disentangle illumination from reflectance through dual-level prior alignment. The architecture consists of two synergistic stages:

Stage 1: Physically Aligned Normalization (PAN)

This is a model-free, closed-form preprocessing module that operates before the neural network to correct global illumination biases. It consists of three steps:

1. Gray-world Normalization: Corrects chromatic bias caused by illuminant color (e.g., warm indoor light) by balancing channel-wise illumination.
2. Log-domain Retinex Decomposition: Transforms the image into the logarithmic domain to separate illumination ($S$) and reflectance ($R$) additively ($\log I = \log R + \log S$). It estimates global lighting via spatial averaging in log-space and isolates the pseudo-reflectance as the residual.
3. Dynamic Range Recombination: Recombines the separated components and normalizes the dynamic range to produce an illumination-consistent input ($\hat{I}$).

• Goal: Suppress global chromatic bias and stabilize luminance distribution without learning parameters.

Stage 2: Geometric-Semantic Rectification Attention (GSRA)

This module operates within the Transformer-based encoder-decoder backbone to align conflicting physical cues.

• Inputs: It integrates DepthAnything-v2 (for geometric priors: depth and normal maps) and DINO-v2 (for semantic embeddings).
• Mechanism: It extends Differential Attention to a cross-modal setting. Instead of simply fusing features, it computes a rectified attention map:
  $$A_{rect} = A_{sem} - \lambda \cdot A_{geo}$$
  Where $A_{sem}$ is semantic attention and $A_{geo}$ is geometric attention.
• Function: This subtraction structure acts as a physically interpretable gate. It uses geometric guidance to rectify semantic attention, suppressing semantic over-smoothing at true illumination boundaries while preserving geometric precision. This resolves the conflict where geometric features are noisy in uniform regions but precise at edges, while semantic features are globally stable but spatially coarse.

3. Key Contributions

1. Physically Aligned Normalization (PAN): A novel, parameter-free preprocessing module that uses closed-form Retinex decomposition to suppress chromatic bias. It improves existing architectures by 0.15–0.34 dB across diverse lighting conditions.
2. Geometric-Semantic Rectification Attention (GSRA): A new attention mechanism that explicitly aligns depth-derived geometry with DINO-v2 semantic embeddings. It harmonizes local geometric precision with global semantic stability, addressing the "modal misalignment" problem in ambient lighting.
3. Robust Generalization: The framework achieves state-of-the-art performance not only on standard single-light shadow removal benchmarks but also generalizes effectively to multi-source ambient lighting normalization (ALN) scenarios where traditional methods fail.

4. Experimental Results

The authors evaluated PhaSR on five datasets: ISTD, ISTD+, INS, WSRD+, and the challenging Ambient6K (multi-source indoor lighting).

• Quantitative Performance:
  • Shadow Removal: PhaSR achieves SoTA results on WSRD+ (28.44 PSNR) and competitive results on ISTD+ (34.48 PSNR).
  • Ambient Normalization: On the difficult Ambient6K dataset, PhaSR outperforms dedicated ALN methods (IFBlend, RLN2-Lf) and other shadow removal models (OmniSR, DenseSR), achieving 23.32 PSNR and 0.834 SSIM.
• Efficiency:
  • PhaSR is highly efficient, requiring only 55.63 G FLOPs and 18.95 M parameters.
  • It runs at 87.9 ms per image (640x480), significantly faster than diffusion-based approaches (e.g., StableShadowDiffusion at 452.8 ms) while maintaining superior quality.
• Cross-Dataset Generalization:
  • In cross-dataset tests (training on one, testing on another without fine-tuning), PhaSR demonstrated superior robustness. For example, when trained on complex indoor lighting (Ambient6K) and tested on outdoor shadows (ISTD), it outperformed OmniSR by +1.46 dB.
• Ablation Studies:
  • Removing PAN caused consistent performance drops, validating its role in stabilizing inputs.
  • Removing GSRA led to significant drops, especially on real-world data, confirming the necessity of cross-modal alignment.
  • The differential rectification mechanism ($A_{sem} - \lambda A_{geo}$) was proven superior to naive fusion.

5. Significance and Impact

• Bridging the Gap: PhaSR successfully bridges the gap between controlled single-light shadow removal and the complex, ill-posed problem of ambient light normalization.
• Physical Grounding: By explicitly modeling the physics of light transport (via Retinex) and aligning geometric/semantic priors, the method avoids the "black box" limitations of purely data-driven approaches.
• Practical Deployment: The model's low computational cost and lack of dependency on explicit shadow masks make it highly suitable for real-time applications and embedded systems.
• Limitations: The paper acknowledges that the method still struggles with intrinsically dark materials (where shadow and material are indistinguishable) and specular surfaces (which violate Lambertian assumptions), suggesting future work in material-aware priors.

In conclusion, PhaSR represents a significant advancement in image restoration by introducing a physically aligned framework that leverages closed-form normalization and cross-modal attention to handle the full spectrum of lighting conditions, from direct shadows to complex ambient environments.