💻 computer science

Cast and Attached Shadow Detection via Iterative Light and Geometry Reasoning

This paper proposes a physically grounded, iterative framework that jointly detects cast and attached shadows by leveraging a closed feedback loop between light estimation and shadow detection modules, achieving significant performance improvements on a newly curated dataset.

Original authors: Shilin Hu, Jingyi Xu, Sagnik Das, Dimitris Samaras, Hieu Le

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shilin Hu, Jingyi Xu, Sagnik Das, Dimitris Samaras, Hieu Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a 3D sculpture in a museum. You can tell it's round and bumpy not just because of its color, but because of the shadows playing across its surface.

For a long time, computers were great at spotting the big, dark shadows objects throw on the floor (like a tree casting a shadow on the grass). But they were terrible at seeing the tiny, subtle shadows that happen on the object itself (like the dark side of a bumpy rock where the light can't reach).

This paper introduces a new "super-vision" system that finally teaches computers to see both types of shadows, using a clever trick inspired by how human brains work.

Here is the breakdown in simple terms:

1. The Problem: The "One-Size-Fits-All" Mistake

Imagine you are trying to clean a dirty window.

Cast Shadows are like a smudge on the outside of the glass. You can wipe them away easily.
Attached Shadows are like the natural darkness on the inside curve of a folded piece of paper. If you try to "wipe" this away with a generic cleaner, you accidentally erase the paper's shape, making it look flat and fake.

Previous computer programs tried to find all shadows with one big net. They caught the big floor shadows well, but they missed the subtle shadows on the objects, or they got confused and erased the object's shape entirely.

2. The Solution: A Detective Duo

The authors built a system with two "detectives" working together in a loop:

Detective A (The Shadow Hunter): Looks at the image and tries to draw the shadow lines.
Detective B (The Light Finder): Looks at the image and guesses where the sun is shining from.

The Magic Loop:

Detective B guesses the light direction.
Using that guess, it draws a rough map of where shadows should be based on physics (if a surface faces away from the sun, it must be in shadow).
Detective A takes this rough map and says, "Ah! Now I see! That dark spot isn't just a shadow; it's the side of the rock facing away from the sun."
Detective A draws a better shadow map and hands it back to Detective B.
Detective B says, "Oh, now that I see the shadows more clearly, I can guess the light direction even better!"

They repeat this conversation over and over, getting smarter with every round, until they perfectly separate the "floor shadows" from the "object shadows."

3. The New Map (The Dataset)

To train these detectives, the researchers needed a special map. They went through three existing photo collections and manually drew two separate outlines for every picture:

The outline of the shadow on the floor.
The outline of the shadow on the object.

Think of it like a coloring book where, instead of just one "shadow" color, they created a specific "floor shadow" color and a specific "object shadow" color. This is the first time a dataset like this has been made available for computers to learn from.

4. Why This Matters: The "Magic Eraser" Test

The paper proves this works by testing it on a Shadow Removal task (making a photo look like it was taken in bright, shadow-free sunlight).

Old Method: Tries to remove all shadows at once. It accidentally makes the object look flat, like a 2D drawing, because it erased the natural shadows that give the object its 3D shape.
New Method: It removes the floor shadows but keeps the attached shadows. The result? The object still looks round, bumpy, and realistic, but the dark floor is gone.

The Bottom Line

This paper teaches computers to understand that shadows aren't just dirt to be cleaned off; they are clues about shape and light. By having the computer guess the light source and the shadows simultaneously, it can finally "see" the 3D world the way we do, distinguishing between a shadow cast by an object and the shadow that lives on the object itself.

1. Problem Definition

Shadow detection is a fundamental computer vision task, yet existing methods suffer from two primary limitations:

Unified Mask Prediction: Most state-of-the-art (SOTA) methods predict a single, unified shadow mask, failing to distinguish between cast shadows (shadows projected onto external surfaces) and attached shadows (shadows formed on the object's own surface due to self-occlusion).
Neglect of Attached Shadows: Many approaches either ignore attached shadows entirely or treat them identically to cast shadows. This is problematic because attached shadows encode critical information about local surface orientation and geometry, whereas cast shadows reveal spatial relationships between objects.
Downstream Impact: The inability to separate these shadow types hinders applications like 3D reconstruction and shadow removal. For instance, applying a standard shadow removal algorithm to a unified mask often over-attenuates object surfaces, leading to unnatural color recovery.

2. Methodology

The authors propose a physically grounded, iterative framework that jointly detects cast and attached shadows by modeling their dependence on light direction and surface geometry.

Core Insight

The approach is built on the physical observation: surfaces facing away from the light source tend to fall into shadow. There is a reciprocal relationship between shadow formation and light estimation:

Given a light direction and surface normals, one can approximate attached shadows.
Conversely, observed shadows constrain the possible light direction.

Framework Architecture

The system consists of two modules operating in a closed feedback loop:

Shadow Detection Module: A multi-class segmentation network (based on SILT [70]) that takes an RGB image and a surface normal map as input. It predicts three classes: Background, Cast Shadow, and Attached Shadow.
Light Estimation Module: A network (based on ConvNeXt-S [29]) that estimates a 3D light direction vector ( $\ell$ ) based on the normal map and predicted shadows.

Iterative Refinement Process

The framework operates over multiple forward passes (iterations):

Step 1: The shadow detector produces an initial prediction.
Step 2: The light estimator uses the current prediction and normals to estimate the light direction $\ell$ .
Step 3 (Geometry-Based Partial Map): Using the estimated $\ell$ and surface normals ( $n$ ), the system generates a partial attached shadow map ( $M_{attached}$ ) based on the dot product condition:
$M_{attached} = \begin{cases} 1, & \boldsymbol{n} \cdot \boldsymbol{\ell} > 0 \\ 0, & \boldsymbol{n} \cdot \boldsymbol{\ell} \le 0 \end{cases}$
Note: This map is "partial" because it does not account for visibility/occlusion blocking, only surface orientation.
Step 4: This partial map is fed back as an additional input channel to the shadow detector for the next iteration, guiding it to refine the full cast and attached masks.
Step 5: The improved shadow masks provide better cues for the light estimator in the subsequent iteration.

Loss Functions

The training objective combines shadow and light losses:

Shadow Loss: Includes a segmentation loss (BCE + Dice) for global shadow detection and a classification loss (Cross-Entropy + Margin Loss) to explicitly separate cast from attached shadows.
Light Loss: Includes a weighted BCE to align the partial attached map with ground truth, an $L_1$ loss to regress the light direction to a heuristic target, and a unit norm constraint.

3. Key Contributions

Joint Framework: A novel architecture that jointly learns shadow segmentation and light estimation, leveraging the physical coupling between illumination and surface geometry.
Iterative Refinement: A dual-module design where lighting and shadow cues mutually reinforce each other through an iterative feedback loop, progressively refining predictions.
New Dataset: The introduction of a curated dataset of 1,458 images (1,166 train, 292 test) sourced from WSRD, SOBA, and CUHK. Crucially, this dataset provides explicit, separate annotations for cast and attached shadows, addressing the lack of such data in previous benchmarks.
Performance: The method achieves state-of-the-art results, particularly in attached shadow detection, which has been historically difficult.

4. Experimental Results

The method was evaluated on the proposed dataset and compared against seven SOTA methods (e.g., SILT, FSDNet, BDRAR).

Quantitative Performance:
- Attached Shadows: The proposed method achieved a 33% reduction in Balanced Error Rate (BER) compared to retrained baselines (dropping from ~20% BER to 12.93%).
- Cast Shadows: Maintained strong performance, achieving the highest F1 score (85.46%) among all methods.
- Full Shadows: Achieved the best overall BER (6.50) and F1 (86.61).
Qualitative Analysis:
- Visual comparisons show that baseline methods often misclassify dark lit regions as attached shadows or fail to capture the geometry-consistent boundaries of attached shadows.
- The proposed method correctly segments attached shadows based on surface orientation, even in regions where baselines fail.
Ablation Studies:
- Geometry: Removing surface normals caused a significant performance drop, confirming normals are essential priors.
- Iterations: Performance improved significantly from 1 to 3 iterations, with diminishing returns beyond 3.
- Loss Terms: Removing any specific loss term (e.g., the margin loss for separating classes) degraded performance, proving the complementarity of the objectives.
Application (Shadow Removal): A user study demonstrated that using separate attached/cast masks for shadow removal resulted in 80.7% preference over using a unified mask, as it prevented over-attenuation of the object's surface.

5. Significance and Limitations

Significance:

This work bridges the gap between physical scene understanding and deep learning by explicitly modeling the physics of light and geometry.
It provides the first standardized benchmark for separate cast and attached shadow detection, enabling future research in 3D perception and realistic image editing.
The iterative, physics-guided approach offers a robust solution for tasks where appearance-based cues alone are insufficient (e.g., low-contrast attached shadows).

Limitations:

Reliance on Normal Maps: The method depends on the accuracy of surface normals generated by a foundation model. Errors in normal estimation (e.g., flipped normals) propagate to incorrect shadow predictions.
Lighting Assumptions: The framework assumes a single dominant directional light source. It may struggle in complex indoor scenes with multiple light sources, soft shadows from area lights, or low-light conditions with strong indirect illumination.

In conclusion, the paper presents a significant advancement in shadow detection by moving beyond unified masks to a physically informed, iterative approach that successfully disentangles cast and attached shadows, leading to superior performance in both detection and downstream applications.