💻 computer science

From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition

This paper proposes repurposing generative inpainting models for image layer decomposition through lightweight fine-tuning and a novel multi-modal context fusion module, achieving superior object removal and occlusion recovery using only a synthetic dataset.

Original authors: Jingxi Chen, Yixiao Zhang, Xiaoye Qian, Zongxia Li, Cornelia Fermuller, Caren Chen, Yiannis Aloimonos

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingxi Chen, Yixiao Zhang, Xiaoye Qian, Zongxia Li, Cornelia Fermuller, Caren Chen, Yiannis Aloimonos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, complex painting, like a portrait of a person standing in front of a sunset. Right now, if you want to move the person to a different background, or if you want to remove the person entirely to see what the sunset looks like without them, you have to do it manually. It's like trying to cut a piece of paper out of a glued-down collage without tearing the paper underneath. It's messy, difficult, and often leaves ugly holes.

This paper introduces a new "magic trick" for computers that solves this problem. Here is the simple breakdown of what they did:

1. The Big Idea: "The Magic Eraser and the Copy-Paste Tool"

The researchers realized that the task of Layer Decomposition (separating the person from the background) is actually very similar to two things computers are already really good at:

Inpainting: Filling in a hole in a picture (like erasing a watermark).
Outpainting: Extending a picture beyond its edges (like imagining what's behind the frame).

They asked a simple question: "If we already have a super-smart AI that can fill in holes and extend pictures, can we just teach it a tiny bit of new math to also separate objects?"

Instead of building a brand-new, expensive super-computer brain from scratch, they took a pre-trained "Inpainting Model" (an AI that already knows how to fix broken pictures) and gave it a "lightweight upgrade."

2. The Analogy: The Master Chef and the New Recipe

Think of the pre-trained Inpainting Model as a Master Chef who has spent 10 years perfecting the art of baking bread. They are amazing at it.

The researchers wanted this chef to also learn how to make a specific type of sandwich (separating the foreground from the background).

The Old Way: Fire the chef, hire a new team, and spend millions of dollars training a whole new kitchen from scratch to learn sandwiches.
This Paper's Way: Take the Master Chef, give them a quick 15-minute tutorial on how to layer the bread, meat, and lettuce, and let them use their existing bread-baking skills to make the sandwich.

They used a technique called LoRA (Low-Rank Adaptation). Imagine this as giving the chef a special, thin "recipe card" to hold in their apron. They don't need to rewrite the chef's entire memory; they just add this small card to guide them on the new task.

3. The Secret Sauce: "The Context Clues"

One of the hardest parts of separating a person from a background is knowing exactly where the person ends and the background begins, especially if the person is holding a cup or if their hair is messy.

The researchers added a Multi-Modal Context Fusion module.

The Metaphor: Imagine you are trying to cut a shape out of a piece of paper. If you only look at the color, you might cut too close. But if you also have a shadow map (depth), a sketch (edges), and a color guide (segmentation) laid out on the table, you can cut perfectly.
The AI looks at the photo plus these extra "maps" (like a depth map showing how far away things are) to understand the 3D shape of the object. This stops the AI from getting confused and "hallucinating" (making up things that aren't there).

4. The Training: "The Frankenstein Dataset"

To teach this AI, they needed thousands of examples of "Person + Background" pairs. But high-quality, real-life examples are rare and expensive.

The Solution: They created a "Frankenstein" dataset. They took real photos (which have great texture but messy edges) and AI-generated photos (which have perfect shapes but look a bit plastic).
They mixed them together. It's like taking a real, delicious steak and mixing it with a perfect, synthetic steak to create a training meal that teaches the AI the best of both worlds: realistic texture and perfect shapes.

5. The Result: "The Layer Cake"

When you run an image through their new system, it doesn't just give you one picture. It gives you a Layer Cake:

The Foreground: The person (or object) is lifted out, complete with their hair and clothes, ready to be pasted onto a new background.
The Background: The person is removed, and the AI instantly "fills in" the missing space with the original sunset, looking like the person was never there.

Why This Matters

It's Cheap: You don't need a supercomputer to run this. It runs on standard hardware because it reuses existing models.
It's Fast: Because they didn't retrain the whole brain, just added a small "recipe card," it's efficient.
It's Creative: It unlocks new possibilities for artists, designers, and editors. You can now easily move objects around, remove distractions, or create new scenes without needing to be a Photoshop wizard.

In short: They took a computer that was already great at fixing holes in pictures, gave it a few extra clues and a tiny new instruction manual, and now it can perfectly separate objects from their backgrounds like a professional editor.

1. Problem Definition

Image Layer Decomposition is the task of separating a single image into distinct layers: a foreground (the object of interest, potentially with occlusions recovered) and a background (the scene with the object removed and the occluded regions filled in).

Challenges:
- Data Scarcity: High-quality, real-world datasets with ground-truth RGBA (Red, Green, Blue, Alpha) layers are extremely rare.
- Computational Cost: Existing state-of-the-art methods (e.g., LAYERDECOMP) often require full fine-tuning of massive text-to-image models on large, curated datasets, which is computationally prohibitive for many researchers and practitioners.
- Task Complexity: The task requires simultaneous inpainting (filling the background where the object was) and outpainting (extending the foreground object to include occluded parts while maintaining transparency).

2. Methodology: "Outpaint-and-Remove"

The authors propose a novel framework that reframes layer decomposition as a combination of inpainting and outpainting tasks. Instead of training a model from scratch, they efficiently adapt a pre-trained Diffusion-based Inpainting Model (specifically FLUX.1-Fill-dev) using lightweight fine-tuning.

Core Architecture & Components

The pipeline modifies a standard Diffusion Transformer (DiT) with three key innovations:

Multi-Modal Context Tokenization (Linear Attention):
- To preserve fine details in the latent space, the model incorporates auxiliary inputs: edge maps, segmentation maps, and depth maps.
- Challenge: Fusing tokens from multiple modalities increases computational complexity quadratically ( $O(K^2)$ ).
- Solution: A Multi-Modal Latent Fusion module inspired by linear attention transformers. It uses a small, constant number of latent tokens ( $N$ ) as queries to summarize the multi-modal context, reducing complexity to linear $O(KN)$.
Dual Image-Mask Context Design:
- Standard inpainting models use only a background context ( $c_{I-M}^b$ ).
- Innovation: The authors introduce a foreground context ( $c_{I-M}^f$ ) alongside the background context.
- Mechanism: The model receives concatenated tokens for both the foreground (object + alpha) and background (scene). The foreground context acts as a control signal to balance preservation of existing object details with generation of occluded parts, preventing the model from hallucinating or altering the object's identity.
Parameter-Efficient Fine-Tuning (PEFT) & Decoding:
- LoRA (Low-Rank Adaptation): The base DiT weights are frozen. Only the input projection layer and LoRA layers inserted into attention/feed-forward blocks are trained.
- Decoding: The background is decoded as standard RGB. The foreground is decoded as RGBA (including an alpha channel for transparency), requiring a separate finetuned RGBA encoder/decoder.

Data Curation Strategy

Since high-quality layered datasets are scarce, the authors constructed a synthetic dataset (100k triplets) by combining:

Real Foregrounds: From the MULAN dataset (rich detail, but often imperfect shapes due to occlusions).
Synthetic Foregrounds: Generated by LayerDiffuse (complete shapes, but smooth/low-fidelity textures).
Backgrounds: Sampled from OpenImages.
Hybrid Approach: Merging real and synthetic sources creates a balanced dataset that teaches the model to handle both realistic textures and complete object geometries.

3. Key Contributions

Novel Perspective: Establishes a theoretical and practical link between layer decomposition and in/outpainting, proving that a single inpainting backbone can handle both tasks with minimal modification.
Efficient Adaptation: Introduces a lightweight, parameter-efficient method (LoRA + Multi-modal fusion) that achieves SOTA performance without large-scale retraining or proprietary data.
Dual-Context Mechanism: Proposes a specific image-mask context design that significantly reduces hallucinations in foreground extraction by explicitly guiding the model on what to preserve vs. what to generate.
Open-Source Reproducibility: The entire training pipeline relies on publicly available datasets (MULAN, OpenImages) and tools, democratizing access to layer decomposition technology.

4. Experimental Results

The model was evaluated on the MULAN dataset and a custom set of real-world high-quality images.

Quantitative Performance (Background Removal):
- Outperformed SOTA baselines (SD-XL Inpainting, PowerPaint, BrushNet, OmniEraser, etc.) across all metrics.
- PSNR: 27.30 (vs. 25.59 for the base FLUX model and ~23.46 for PowerPaint).
- FID: 25.97 (lower is better), significantly outperforming the base model (35.96).
Quantitative Performance (Foreground Extraction):
- Compared against matting baselines (MattingAnything, DiffMatte).
- Achieved the highest User Preference Rate (59.51%) in a study with 18 researchers, indicating superior visual fidelity and practical utility.
Ablation Studies:
- Context: Removing the foreground context ( $c_{I-M}^f$ ) caused a 0.26 dB drop in PSNR and significant hallucination of object details.
- LoRA Rank: A rank of 256 was optimal. Rank 128 was insufficient for learning the new task; Rank 1024 overwrote pre-trained priors, causing hallucinations.
- Multi-modal: Removing depth/edge/segmentation cues degraded performance, confirming their role in semantic understanding.

5. Significance and Impact

Democratization: By avoiding the need for massive, proprietary datasets and full-model retraining, this method makes advanced layer decomposition accessible to a broader range of users and researchers.
Efficiency: It demonstrates that "repurposing" existing generative models via lightweight adaptation is a viable and superior strategy for complex multi-task problems compared to training from scratch.
Applications: The ability to simultaneously extract clean foregrounds (with occlusion recovery) and reconstruct backgrounds unlocks new possibilities for:
- Creative re-composition and layered artwork generation.
- Component-based image retrieval.
- Advanced video editing and object manipulation.

In conclusion, the paper successfully bridges the gap between generative inpainting and layer decomposition, offering a highly efficient, data-scarce-friendly solution that sets a new standard for image editing capabilities.