💻 computer science

RevealLayer: Disentangling Hidden and Visible Layers via Occlusion-Aware Image Decomposition

This paper introduces RevealLayer, a diffusion-based framework equipped with specialized attention and adapter modules, alongside a new high-quality dataset and benchmark, to achieve precise disentanglement of hidden and visible layers in complex natural images.

Original authors: Binhao Wang, Shihao Zhao, Bo Cheng, Qiuyu Ji, Yuhang Ma, Liebucha Wu, Shanyuan Liu, Dawei Leng, Yuhui Yin

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Binhao Wang, Shihao Zhao, Bo Cheng, Qiuyu Ji, Yuhang Ma, Liebucha Wu, Shanyuan Liu, Dawei Leng, Yuhui Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photograph of a busy street scene. In this picture, a person is standing in front of a car, and the car is parked in front of a shop. To a computer, this is just one flat image. But to a human, we understand that these are three separate "layers" stacked on top of each other.

The paper introduces a new AI tool called RevealLayer that acts like a digital "peeling" machine. Its job is to take that single, flat photo and separate it back into its individual layers (the person, the car, the shop) so you can edit them independently.

Here is how it works, broken down into simple concepts:

The Problem: The "Spaghetti" Mess

Previous AI tools tried to do this by peeling the layers off one by one, like taking apart a sandwich.

First, they find the person.
Then, they try to guess what the background looks like behind the person.
Then, they find the car.

The problem is that if the first step makes a tiny mistake (like leaving a tiny bit of the person's shadow on the background), that mistake gets passed down to the next step. By the time they finish, the image is full of "ghosts," blurry edges, and weird artifacts. It's like trying to untangle a knot of spaghetti by pulling on one strand; the whole thing just gets messier.

The Solution: The "Simultaneous Peeler"

RevealLayer changes the game. Instead of peeling layers one by one, it looks at the whole picture and tries to separate all the layers at the same time.

Think of it like a magician pulling multiple colored scarves out of a single hat all at once, rather than trying to pull them out one by one and hoping the hat doesn't get tangled.

To make this work, the researchers built three special "tools" inside the AI:

The "Zone Keeper" (Region-Aware Attention):
Imagine you are in a crowded room, and you need to listen only to your friend, ignoring everyone else. This tool tells the AI, "Focus only on the pixels inside this specific box (the person) and ignore the pixels from the car." It prevents the AI from getting confused and mixing up the features of different objects.
The "Context Detective" (Occlusion-Guided Adapter):
Sometimes, part of an object is hidden behind another object (occlusion). If the person is blocking the car, the AI has to guess what the hidden part of the car looks like. This tool acts like a detective that looks at the surrounding clues (the visible parts of the car and the background) to intelligently "fill in the blanks" for the hidden parts, ensuring the car looks complete even where it was covered up.
The "Sharpener" (Composite Loss):
When you separate layers, the edges can sometimes get blurry or fuzzy. This tool acts like a high-precision ruler and eraser. It forces the AI to draw very sharp, clean lines between the layers and makes sure there are no leftover "smudges" of the person left on the background.

The New "Training School" (RevealLayer-100K)

To teach this AI how to do such a difficult job, the researchers realized they needed a massive library of examples. Existing libraries were too small or only had simple cartoon drawings.

So, they built RevealLayer-100K.

How they made it: They used a team of automated robots (AI algorithms) to find pictures, cut out objects, and guess the layers. Then, they had human experts review the work to make sure it was perfect.
The Result: A massive dataset of 100,000 complex, real-world photos where every single layer is perfectly labeled. They also created a "test exam" called RevealLayerBench to see how well the AI performs on tricky, messy scenes.

The Results

When they tested RevealLayer against other top methods:

Cleaner Backgrounds: When they removed an object, the background looked natural, without weird shadows or "ghost" shapes.
Sharper Edges: The lines where objects meet the background were crisp, not blurry.
Better "Inpainting": When an object was hidden behind another, RevealLayer did a better job of guessing and reconstructing the hidden parts.

In short, RevealLayer is a new way for computers to understand that a photo is actually a stack of transparent sheets. By looking at the whole stack at once and using special tools to keep the layers separate and the hidden parts filled in, it creates much cleaner, more editable images than ever before.

Technical Summary: RevealLayer

Problem Statement

Recent diffusion-based models have advanced image generation but remain primarily designed for single-layer RGB output, leaving multi-layer image modeling largely unexplored. Decomposing a complex natural image into a background and multiple transparent RGBA foreground layers is challenging due to the need to handle complex occlusions, capture object hierarchies, and complete missing content while maintaining consistency across visible regions.

Existing approaches face significant limitations:

Cascaded Pipelines: Methods that sequentially perform segmentation, alpha matting, and inpainting suffer from error accumulation, leading to accumulated artifacts and degraded layer consistency, especially in heavily occluded regions.
End-to-End Diffusion Models: While frameworks like LayerD and OmniPSD mitigate error accumulation by jointly predicting layers, they often lack explicit user guidance, offering limited control over layer semantics and ordering. Other methods (e.g., Qwen-Image-Layered, CLD) struggle with ambiguous layer semantics, residual artifacts, or are restricted to stylized images rather than complex natural scenes.
Data Scarcity: The field is hindered by a lack of high-quality, large-scale multi-layer natural image datasets, which limits the training and evaluation of robust decomposition models.

Methodology

The authors propose RevealLayer, a diffusion-based framework that decomposes an input RGB image into multiple RGBA layers under user-specified bounding-box guidance. The framework is built upon the FLUX.1 [dev] architecture (utilizing MM-DiT as the backbone) and incorporates three key technical components:

1. Region-Aware Attention (RAA)

To address the issue of inter-layer feature interference in standard self-attention mechanisms, RevealLayer introduces a Region-Aware Attention module. This module imposes a structural constraint on cross-region token interactions via an attention mask.

Mechanism: The mask $M$ governs interactions between query tokens $q$ and key tokens $k$ , ensuring that tokens from the text prompt and global image can interact with all layers, but tokens from a specific layer $L_i$ primarily attend to their own layer and the spatially corresponding region $R_i$ of the global image.
Goal: This promotes region-consistent attention, suppresses cross-layer feature mixing, and enables robust layer-wise disentanglement.

2. Occlusion-Guided Adapter (OGA)

While RAA separates layers, generating visually coherent content within overlapping regions remains difficult. The OGA module enhances semantic coherence by incorporating localized information from the original image.

Mechanism: After the DoubleAttnBlock, the OGA concatenates conditional image features ( $z_c$ ) with per-layer latent representations ( $z_t^i$ ) along the channel dimension. It applies self-attention using a layer-specific mask $M_i$ to enhance semantic coherence, yielding an updated latent representation.
Goal: To leverage contextual information for better reconstruction of occluded and overlapping regions.

3. Composite Loss Function

To enforce sharp boundaries and suppress residual artifacts, the training objective combines three loss terms:

Flow Matching Loss ( $L_{FM}$ ): The standard objective for the diffusion process.
Hard-Constraint Alpha Loss ( $L_{\alpha}$ ): A focal-style penalty applied to the alpha channel to refine foreground generation and ensure pixel-level boundary precision. It focuses on hard boundary pixels to encourage sharper edges.
Soft-Constraint Orthogonality Loss ( $L_{orth}$ ): A cosine-based loss in pixel space designed to suppress undesired inter-layer interactions and target-related artifacts in background reconstruction by discouraging low-frequency, structurally correlated foreground residuals.

4. Dataset Construction (RevealLayer-100K & RevealLayerBench)

To support training and evaluation, the authors constructed:

RevealLayer-100K: A large-scale dataset of natural-scene images with high-quality RGBA layer annotations. It was created via a pipeline involving automated algorithms (Qwen3-VL, Florence-2, SAM-H, Qwen-Image-Edit, ViTMatte) and rigorous human annotation/manual review to ensure background consistency and occlusion recovery fidelity.
RevealLayerBench: A benchmark for systematic evaluation on complex multi-layer layouts.

Key Contributions

RevealLayer Framework: A controllable, bounding-box-guided diffusion framework for decomposing natural images into RGBA layers, capable of handling complex occlusions and maintaining layer consistency.
Occlusion-Aware Paradigm: The integration of Region-Aware Attention, an Occlusion-Guided Adapter, and a composite loss function to disentangle visible and hidden content and recover occluded regions effectively.
Dataset and Benchmark: The introduction of RevealLayer-100K and RevealLayerBench, addressing the scarcity of high-quality multi-layer natural image data and providing a standard for systematic evaluation.

Experimental Results

Extensive experiments demonstrate that RevealLayer outperforms existing approaches across multiple tasks:

Object Removal: On OBER-Test and RevealLayerBench, RevealLayer achieves state-of-the-art background reconstruction metrics (highest PSNR and SSIM, competitive LPIPS and FID) compared to methods like PowerPaint, SmartEraser, and OmniPaint. It effectively eliminates residual artifacts and hallucinated regions.
Image Matting: On AIM-500 and RefMatte-RW100, the method achieves superior performance in alpha channel accuracy (lowest MSE, MAD, SAD) compared to general segmentation models (SAM variants) and specialized matting models (MAM), balancing fidelity, edge sharpness, and content consistency.
Multi-Layer Decomposition: On RevealLayerBench, RevealLayer significantly outperforms Qwen-Image-Layered and CLD in background and foreground disentanglement, generation quality, and alpha accuracy. It achieves higher Soft IoU and superior scores in Q-Insight metrics (Consistency, Fidelity, Editability).
Robustness: The model demonstrates reasonable robustness to bounding-box perturbations (e.g., slight offsets or size variations), though performance degrades with severe inaccuracies.

Significance

The paper claims that RevealLayer represents a significant step forward in controllable, occlusion-aware layer decomposition for real-world natural scenes. By moving beyond cascaded pipelines and unguided end-to-end models, it offers a flexible and robust solution that enables precise layer separation and reliable recovery of occluded content. The work establishes a new benchmark and dataset, facilitating future research in multi-layer image modeling and downstream applications such as object removal, matting, inpainting, and layer-based image editing. The authors acknowledge that challenges remain in cases of severely inaccurate bounding boxes, heavy occlusion, transparent regions, and dense repetitive textures, identifying these as areas for future work.