💻 computer science

Single Image Reflection Removal with Patch Reflectance Prior

This paper proposes a novel approach for single image reflection removal that leverages a general reflection intensity prior, learned via a Reflection Prior Extraction Network (RPEN) and integrated into a Prior-based Reflection Removal Network (PRRN) using a transformer U-Net, to achieve state-of-the-art performance on real-world benchmarks.

Original authors: Dongshen Han, Heechan Yoon, Hyukmin Kwon, Hyun-Cheol Kim, Hyon-Gon Choo, Seungkyu Lee, Chaoning Zhang

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongshen Han, Heechan Yoon, Hyukmin Kwon, Hyun-Cheol Kim, Hyon-Gon Choo, Seungkyu Lee, Chaoning Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a photo of a beautiful landscape through a window. Unfortunately, the window is dirty, and the reflection of your own room (and maybe a lamp or a person) is superimposed over the view outside. The result is a messy, confusing image where you can't tell what's real and what's just a reflection. This is the problem the paper tackles: Single Image Reflection Removal (SIRR).

Here is a simple breakdown of how the authors solved this, using everyday analogies.

The Problem: The "Messy Window"

Usually, computers try to fix this by guessing specific rules. For example, they might assume, "Reflections are always blurry," or "Reflections always look like ghosts."

The Flaw: In the real world, reflections aren't always blurry or ghostly. Sometimes they are sharp and clear. When the computer relies on a specific rule (like "it must be blurry"), it fails when it sees a sharp reflection. It's like trying to find a lost key by only looking under the lamp; if the key is under the sofa, you won't find it.

The Solution: A New Way to Measure "Messiness"

Instead of guessing what the reflection looks like (blurry, ghostly, etc.), the authors decided to measure how strong the reflection is in different parts of the picture. They call this the "Reflection Intensity Prior."

Think of the image not as one big photo, but as a pizza cut into 49 small slices (patches).

The Old Way: Trying to guess the whole pizza's flavor at once.
The New Way: Tasting each tiny slice individually. Some slices might be 90% reflection (mostly your room's reflection), while others are 90% transmission (mostly the view outside).

The Two-Step Machine

The authors built a system with two main parts, working together like a detective team:

1. The Detective: Reflection Prior Extraction Network (RPEN)

This is the "eye" of the operation.

What it does: It looks at the messy window photo and analyzes each of the 49 pizza slices. It asks, "How much of this slice is reflection?"
How it learns: It was trained on a massive library of photos to understand what a "normal" picture looks like. When it sees a slice that looks weird or "cluttered" compared to a normal picture, it flags it as having a strong reflection.
The Output: It creates a map (a "heat map") showing exactly where the reflection is strong and where it is weak.

2. The Cleaner: Prior-based Reflection Removal Network (PRRN)

This is the "hand" that does the cleaning.

What it does: It takes the messy photo and the map from the Detective.
How it works: Instead of blindly scrubbing the whole image, it uses the map to be smart. If the map says, "This slice is 90% reflection," the Cleaner aggressively removes the reflection. If the map says, "This slice is mostly the view outside," the Cleaner leaves it alone.
The Architecture: They used a "Transformer U-Net," which is a fancy name for a smart, efficient cleaning structure that is very good at connecting different parts of an image (like how a transformer connects words in a sentence).

Why This Works Better

The paper tested this on real-world photos (not just fake computer-generated ones).

The "Sharp" Reflection Test: Many old methods fail when the reflection is sharp and clear (like a mirror). Because the authors didn't assume the reflection was blurry, their system handled sharp reflections perfectly.
The "Patch" Advantage: By breaking the image into small pieces, the system realized that reflections aren't uniform. A reflection might be strong in the top-left corner but weak in the bottom-right. Treating the whole image as one big block misses these details.

The Results

When they tested their system against the best existing methods:

It produced clearer, sharper images.
It worked better on difficult scenarios where reflections were strong or sharp.
It achieved the "State-of-the-Art" (the best possible score) on standard tests.

Summary

Imagine you are trying to listen to a friend talk in a noisy room.

Old methods tried to filter out all noise, assuming the noise sounded like a specific hum. If the noise was a shout, the filter failed.
This new method puts a microphone on different parts of the room. It figures out exactly where the shouting is coming from (the "Reflection Intensity") and only mutes those specific spots, leaving your friend's voice clear everywhere else.

The paper claims this approach is more flexible and accurate because it focuses on how strong the reflection is in specific areas, rather than guessing what kind of reflection it is.

Technical Summary: Single Image Reflection Removal with Patch Reflectance Prior

Problem Statement
Single Image Reflection Removal (SIRR) aims to recover a clean transmission image ( $T$ ) from a single input image ( $I$ ) where the scene is superimposed with a reflection ( $R$ ). This is formulated as $I = g(T) + f(R)$ , where $g(\cdot)$ and $f(\cdot)$ represent diverse degradations such as blur, light absorption, refraction, and depth of field effects. The task is ill-posed due to unknown variables and the lack of ground truth in real-world scenarios.

Existing methods often rely on specific prior assumptions (e.g., image gradient sparsity, ghosting cues, or relative smoothness) to separate the layers. However, these assumptions frequently fail in complex real-world environments where reflection characteristics vary significantly (e.g., sharp vs. blurry reflections, varying object distances, and natural illumination). Furthermore, the scarcity of real-world data has led to an over-reliance on synthetic datasets, causing models to lack robustness when applied to diverse, unstructured real-world scenes.

Methodology
The authors propose a framework that avoids rigid, specific prior assumptions in favor of a general Reflection Intensity Prior. This prior defines the strength of the reflection phenomenon in a local region rather than assuming a specific type of degradation (like blur or ghosting). The framework consists of two main components:

Reflection Intensity Prior Definition:
Instead of the standard ratio of reflection to transmission (which can yield infinite values for strong reflections), the authors redefine reflection intensity as a normalized score between 0 and 1:
$\text{ReflectionIntensity} = \frac{\text{Mean}(R)}{\text{Mean}(R) + \text{Mean}(T)}$
This metric is calculated per regional patch to account for the non-uniform nature of reflections in superimposed images.
Reflection Prior Extraction Network (RPEN):
RPEN is designed to estimate the reflection intensity prior for each patch of an input image.
- Architecture: It utilizes a ResNext101 backbone pre-trained on ImageNet to understand regular, non-reflected images.
- Training: The network is fine-tuned on a dataset comprising synthetic images (created by superimposing reflections from CEILNet onto PASCAL VOC images) and real-world images from the CDR dataset.
- Refinement: A cascaded approach using Atrous Spatial Pyramid Pooling (ASPP) is employed on high-level features to capture global relationships between patches and refine the intensity estimation for each of the 49 ( $7 \times 7$ ) patches.
Prior-based Reflection Removal Network (PRRN):
PRRN utilizes the estimated reflection intensity prior to adaptively remove reflections.
- Architecture: It employs a simple Transformer U-Net structure, known for success in image restoration tasks.
- Prior Embedding: The reflection intensity prior for each patch is encoded into features using a sinusoidal embedding mechanism (similar to positional encoding in transformers).
- Adaptive Processing: The encoded prior features are injected into the network via Feature Wise Affine (FWA) layers within ResBlocks. This guides the network to focus on reflection regions and remove noise region-adaptively during the upsampling and downsampling processes.

Key Contributions

General Reflection Intensity Prior: The paper introduces a prior definition free from specific assumptions about reflection types (e.g., blur or ghosting), focusing instead on the local strength of the reflection phenomenon.
RPEN and PRRN Framework: The authors propose a two-stage pipeline where RPEN extracts regional reflection intensity priors, and PRRN utilizes these priors within a Transformer U-Net architecture to perform the removal.
Patch-Based Analysis: The study demonstrates that segmenting images into patches allows for the learning of non-uniform reflection priors, which is crucial for handling diverse real-world scenarios.
Dataset Creation: A reflection intensity prior dataset was created to train the RPEN module.

Experimental Results
The method was evaluated on the CDR (Categorized, Diverse, and Real-world) dataset, as well as Real20, SIR2, and Nature benchmarks.

Performance: The proposed PRRN+RPEN achieved State-of-the-Art (SOTA) accuracy across most benchmarks. On the CDR test set, it achieved a PSNR of 24.31 (vs. 22.04 for the previous best YTMT-UCS) and an SSIM of 0.815.
Robustness: The method showed exceptional performance in the SRST (Sharp Reflection, Sharp Transmission) category, where many prior-based methods fail due to the lack of specific cues like ghosting. It also performed well in BRST (Blurry Reflection) and strong reflection scenarios.
Ablation Studies:
- Patch Resolution: Experiments showed that while training RPEN with smaller patches (higher resolution) is more challenging, using finer-grained prior truth as input to PRRN significantly improves removal accuracy.
- RPEN Impact: Adding the RPEN module to PRRN resulted in a 0.87 dB increase in PSNR. Visual analysis using LayerCAM confirmed that RPEN guides the network to focus attention on actual reflection regions, whereas PRRN without RPEN tends to focus on non-reflection areas.

Significance and Claims
The paper claims that its primary significance lies in addressing the limitations of existing SIRR methods that rely on specific, often brittle, prior assumptions. By defining a general reflection intensity prior and learning it via RPEN, the proposed approach achieves robustness in diverse real-world scenarios where reflection characteristics are unpredictable. The authors note that while their approach achieves SOTA accuracy, the prediction accuracy of the reflection intensity prior in certain cases could potentially be improved in the future through multi-resolutional RPEN designs. The work emphasizes that region-adaptive reflection removal, guided by learned intensity priors rather than fixed assumptions, is a critical step forward for real-world image restoration.