🤖 AI

SIRR-LMM: Single-image Reflection Removal via Large Multimodal Model

This paper introduces SIRR-LMM, a novel approach that combines a physically accurate synthetic dataset generation framework with a fine-tuned Large Multimodal Model to achieve state-of-the-art single-image reflection removal and separation performance.

Original authors: Yu Guo, Zhiqiang Lao, Xiyun Song, Yubin Zhou, Heather Yu

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Guo, Zhiqiang Lao, Xiyun Song, Yubin Zhou, Heather Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Ghost" in the Window

Imagine you are trying to take a photo of a beautiful painting hanging inside a museum, but the glass case in front of it is reflecting the people and lights behind you. The camera sees a messy mix: the painting (what you want) and the reflection (what you don't want).

In the world of computer vision, this is called Single-Image Reflection Removal (SIRR). It's a tricky puzzle because the camera only has one picture to work with. It's like trying to separate two different songs that were recorded on the exact same microphone at the same time.

For a long time, computers struggled with this because:

Real data is hard to get: To teach a computer how to fix this, you need "answer keys" (photos showing the painting without the reflection and the reflection without the painting). Taking these photos in the real world is nearly impossible without moving the glass or the painting.
Fake data is too fake: Previous attempts made "fake" training data by just layering two photos on top of each other. But real glass doesn't just sit on top of an image; it bends light, creates blurry ghost images, and changes colors based on the angle. Simple layering misses all that physics.

The Solution: A "Physics-Perfect" Simulator

The authors built a new way to create training data that acts like a super-accurate video game simulator.

Instead of just stacking photos, they used a technique called path tracing. Imagine a ray of light as a tiny billiard ball. The simulator shoots millions of these "light balls" at a 3D model of a glass window.

Some balls bounce off the glass (creating the reflection).
Some balls go through the glass (creating the view of the painting).
Some bounce around inside the glass before coming out (creating those blurry, double-image "ghosts").

They combined these physics simulations with real photos of the world. The result is a dataset where the computer learns exactly how light behaves in real life, including tricky things like overexposed bright spots or blurry reflections.

The Brain: Teaching a "Multilingual" Artist

The second part of their invention is how they teach the computer to solve the puzzle. They didn't build a new computer from scratch; instead, they used a Large Multimodal Model (LMM).

Think of an LMM like a super-artist who has seen billions of pictures and can describe them in words. This artist already knows what glass looks like because they've seen so many photos of windows.

The authors used a clever trick to teach this artist a specific job:

The "Sandwich" Method: Instead of showing the artist the messy photo and asking for a clean one, they fed the model a "sandwich" image. They pasted the messy photo, the clean painting, and the reflection side-by-side into one giant image.
The Secret Recipe (LoRA): They didn't retrain the whole massive artist (which would take forever and cost a fortune). Instead, they added a small, lightweight "adapter" (called LoRA) to the artist's brain. This adapter taught the model: "When you see this specific sandwich of images, learn the pattern of how to separate them."

They also gave the model a specific "recipe card" (a text prompt) that explained the task: "Here is a photo with glass, here is the view through it, and here is the reflection." By training on this specific recipe, the model learned the logic of reflection removal without needing to be told about it every single time it works.

The Results: Cleaner Windows, Sharper Reflections

When they tested this new method against the best existing tools:

It sees better: In photos where the reflection was so bright it washed out the image (overexposed), their method could "inpaint" (fill in) the missing details correctly. For example, if a reflection showed a red stand, the model knew to keep the red color in the reflection layer, even if the main image was washed out.
It separates better: While other methods often left "ghosts" or blurry smudges behind, this method produced a much cleaner view of the object behind the glass.
It recovers the reflection: Most tools just try to delete the reflection. This tool actually reconstructs the reflection as a separate, clear image. It can guess what the reflection looks like even in dark areas where the reflection is barely visible, using its knowledge of how light works.

Summary

In short, the authors solved the "dirty window" problem by:

Building a physics-based simulator to create perfect training data (so the computer learns the laws of light, not just patterns).
Using a pre-trained "super-artist" AI and giving it a tiny, specialized upgrade (LoRA) to learn how to separate the "ghost" from the "real thing."

The result is a system that can look at a single photo of a window and magically split it into two perfect images: one showing what's behind the glass, and one showing exactly what is reflected on it.

Technical Summary: SIRR-LMM

Problem Statement

Single-image reflection removal (SIRR) is a challenging computer vision task due to the inherent ill-posed nature of separating tightly coupled reflected and transmitted light, particularly in overexposed regions. While deep learning has advanced the field, progress is bottlenecked by the scarcity of high-quality training data. Real-world datasets are difficult to construct because capturing accurate ground truth (separating reflection and transmission) is physically demanding. Conversely, existing synthetic datasets often rely on simple image blending or simplified reflection models that fail to capture the physical realism required for robust learning, such as complex light transport, refraction, and attenuation.

Methodology

1. Physically Accurate Synthetic Dataset Generation

The authors propose a novel framework to generate a high-fidelity synthetic dataset that overcomes the limitations of traditional image blending.

Path-Traced Rendering: Instead of mixing 2D images, the framework utilizes path tracing on 3D glass models to simulate physically accurate light transport. This involves solving the rendering equation for non-emissive transparent surfaces, accounting for Bidirectional Reflectance Distribution Functions (BRDF), refraction, and multiple light bounces (transmission $T$ and reflection $R$ ).
Hybrid Scene Setup: To avoid the complexity of modeling full 3D scenes from scratch, the method combines High Dynamic Range (HDR) environment maps with real-world RGB images.
- Lighting: HDR maps are used to simulate realistic lighting and highlights, preventing the overexposure issues common in Low Dynamic Range (LDR) based synthesis.
- Variations: The framework supports diverse configurations, including HDR-HDR, HDR-LDR (photographer in reflection), and LDR-HDR (artwork behind glass), to mimic real-world scenarios like museums or outdoor photography.
Parameterization: The system systematically varies glass properties (thickness, color, Index of Refraction, roughness) and camera parameters (aperture, focal distance, field of view) to generate diverse depth-of-field and defocus effects.
Post-Processing: Rendered images undergo gamma correction, auto white balance, and JPEG compression artifacts to mimic real-world imaging conditions.

2. Large Multimodal Model (LMM) Fine-Tuning

The paper leverages the generative capabilities of the FLUX.1 Kontext model, a state-of-the-art text-to-image model based on Diffusion Transformers (DiT).

In-Context LoRA: Rather than full-parameter training, the authors employ Low-Rank Adaptation (LoRA) to fine-tune the model. This allows the model to learn the specific task of reflection removal while preserving its broad pre-trained knowledge.
Joint Captioning and Concatenation: The input image ( $I$ ), transmission layer ( $T$ ), and reflection layer ( $R$ ) are concatenated into a single composite image $[I:T:R]$ . A unified prompt describes the entire decomposition task, implicitly maintaining the correlation between the layers.
Training Strategy: The prompt is embedded as a task-specific feature during training. During inference, the prompt is fixed, allowing the model to function as a single-image reflection removal system without requiring user-provided text descriptions. The input image $I$ is also used as a control image to enhance the correlation between the conditioning and generated outputs.

Key Contributions

Synthetic Data Framework: Introduction of a path-traced dataset generation framework that combines HDR environment maps with real imagery to create physically accurate reflection scenarios, avoiding the pitfalls of simplified blending models.
LMM Application: To the best of the authors' knowledge, this is the first work to explore fine-tuned Large Multimodal Models for single-image reflection removal, demonstrating that foundation models trained on billions of images possess prior knowledge of transparent materials.
Performance: Empirical validation showing that an LMM adapted with task-specific LoRA and high-fidelity synthetic data achieves superior performance compared to state-of-the-art specialized neural architectures.

Results and Analysis

Quantitative Performance

The method was evaluated on three standard real-world benchmarks: Real, Nature, and SIR2.

Metrics: Performance was measured using PSNR, SSIM, and LPIPS.
Findings: The proposed method ("Ours (syn)") achieved competitive results, often outperforming or matching state-of-the-art methods (such as RDNet and DSIT) across all datasets. Notably, the method achieved these results using only a small amount of high-quality synthetic data (1,000 image pairs) and reduced training time (4,000 steps on a single GPU).
Regional Analysis: When evaluating LPIPS specifically within the reflection regions (rather than the whole image), the method showed an 18.2% improvement over competitors, highlighting its effectiveness in handling the most difficult parts of the image.

Qualitative Comparison

Removal: Visual comparisons demonstrate that the method produces cleaner transmission layers with fewer residual artifacts, particularly in overexposed regions and areas with complex occlusions. It successfully inpaints occluded areas and corrects color shifts (e.g., identifying red reflections).
Separation: Unlike previous methods that struggle with informative reflection content, the proposed approach reconstructs meaningful reflection scenes even under low-visibility conditions, adhering to optical principles like Snell's law.
Robustness: The method generalizes well to real-world captures not present in the training set, including smartphone photos and complex museum/gallery scenarios.

Significance and Claims

The paper claims that the primary bottleneck in reflection removal is the lack of high-quality, physically realistic training data. By addressing this through a path-traced synthetic framework, the authors demonstrate that foundation models (LMMs) can be effectively adapted for specialized vision tasks with limited data.

The significance lies in shifting the paradigm from designing complex, task-specific neural architectures to leveraging the inherent priors of large-scale generative models. The authors assert that their approach not only improves reflection removal and separation performance but also provides a scalable pathway for training on physically accurate data without the prohibitive cost of capturing real-world ground truth. The work suggests that with appropriate adaptation (LoRA) and high-fidelity data, general-purpose models can outperform specialized systems in handling complex visual phenomena like glass reflections.