💻 computer science

ClickRemoval: An Interactive Open-Source Tool for Object Removal in Diffusion Models

ClickRemoval is an open-source, interactive tool that enables precise object removal and natural background restoration in complex scenes using only user clicks and pretrained Stable Diffusion models, eliminating the need for manual masks, text prompts, or additional training.

Original authors: Ledun Zhang, Yatu Ji, Xufei Zhuang, Xinying Yao

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ledun Zhang, Yatu Ji, Xufei Zhuang, Xinying Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photo with an unwanted guest in it—maybe a stranger in the background, a trash can in a beautiful landscape, or a messy object on your dining table. Usually, to get rid of them, you have to use photo editing software to carefully "paint over" the object with a brush, tracing its outline pixel by pixel. This is like trying to cut a specific shape out of a piece of paper with a dull knife; it takes a long time, requires a steady hand, and if you miss a spot, the cut looks jagged and ugly.

ClickRemoval is a new, free tool that changes the game. Instead of painting a mask, you just click on the object you want to remove. That's it. One click, and the computer figures out exactly what to delete and what to keep, then magically fills in the empty space with a realistic background.

Here is how it works, using some simple analogies:

1. The "Click-to-Map" Translator (M2N2)

When you click on an object, the tool doesn't just see a dot; it understands the concept of that object.

The Analogy: Imagine you drop a pebble into a pond. The ripples spread out, showing you exactly how far the water reaches. ClickRemoval does something similar. It uses a "ripple effect" (called a semantic distance map) to figure out exactly where the object ends and the background begins, even if the object has a weird shape or is partially hidden. It turns your single click into a detailed map of the object without you having to draw a single line.

2. The "Attention Switch" (SGAR & SGAS)

Once the tool knows what the object is, it needs to tell the AI model, "Ignore this part, and focus on the background."

The Analogy: Think of the AI model as a painter who is very focused on the whole picture. Usually, the painter looks at everything equally. ClickRemoval acts like a spotlight operator.
- SGAR (The Spotlight): It dims the lights on the object you clicked (telling the painter, "Don't paint this") and brightens the lights on the background (telling the painter, "Fill this in with what belongs here").
- SGAS (The Dimmer Switch): If you keep the lights dimmed on the object the whole time, the painting might look weird or unnatural. So, this part acts like a dimmer switch. It turns the "ignore" signal on strong at the beginning, then slowly fades it out as the painting gets closer to finished. This ensures the background looks natural and not forced.

3. The "Blending Chef" (ARG)

Finally, the tool needs to decide how much of the "new" background to mix with the "old" original image.

The Analogy: Imagine you are making a smoothie. You have the original fruit (the background) and a new flavor (the AI's guess). ARG is the chef who decides the perfect ratio. If you want a result that looks very close to the original photo, the chef adds a little new flavor. If you want a complete transformation, the chef adds more. This gives you control over how strong the removal effect is.

Why is this a big deal?

Most other tools are like professional chefs who require a specific recipe (text prompts) and a pre-prepared ingredient list (manual masks) before they can cook. They also often need to go to culinary school (additional training) to learn how to handle specific ingredients.

ClickRemoval is like a smart kitchen assistant that works with any recipe you already have.

No Training Needed: It works right out of the box with existing AI models. You don't need to teach it anything new.
No Masks or Text: You don't need to draw outlines or type "remove the dog." Just click.
Open Source: The creators have shared the entire "recipe book" (code, instructions, and tools) for free, so anyone can use it or build upon it.

The Results

The creators tested this tool against other top-tier photo editors.

Quality: In tests, the photos it produced looked just as good as, or better than, the complex methods that require masks and text.
User Preference: When regular people (non-experts) looked at the results, they preferred ClickRemoval the most. They found it easier to use and the results looked more natural.
Versatility: It works on different types of photos, from simple 512x512 images to high-definition 1024x1024 images, and even handles tricky situations like removing a dog that is hiding behind a fence (using "negative clicks" to tell the tool, "Don't remove the fence, just the dog").

In short, ClickRemoval takes the hard work of photo editing and turns it into a simple "point-and-click" experience, making professional-level object removal accessible to anyone with a computer.

Technical Summary: ClickRemoval

Problem Statement

Existing object removal tools for multimedia content creation and privacy protection often present significant barriers for non-expert users. Current state-of-the-art methods, such as AttentiveEraser, PowerPaint, and the SD-Inpaint series, typically rely on fine-grained manual masks, text prompts, or specialized training pipelines. These dependencies result in high interaction costs and limit usability in complex scenes, often leading to incomplete object removal or unnatural background completion. There is a need for a solution that enables precise object removal without requiring manual masking, textual descriptions, or additional model training.

Methodology

ClickRemoval is an open-source, interactive object removal tool built upon pretrained Stable Diffusion models (SD1.5, SD2.1, and SDXL1.0). It operates solely on user clicks, eliminating the need for hand-drawn masks or text prompts. The system functions by converting user clicks into soft semantic maps and utilizing these maps to redirect self-attention mechanisms within the diffusion model during the denoising process.

The framework consists of three primary components:

M2N2 Semantic Distance Map Extraction:
- User clicks are converted into semantic distance maps using the M2N2 approach. This involves a single denoising forward pass on a frozen Stable Diffusion model to aggregate multi-head self-attention maps into a transition matrix.
- Markov propagation is performed starting from the clicked position to determine the semantic distance of each pixel.
- Object Map ( $M_{ob}$ ): Generated by applying Flood Fill to the semantic map to suppress local minima and enhance instance awareness, defining the region to be removed.
- Background Map ( $M_{bg}$ ): A normalized Markov map (without Flood Fill) that measures semantic distance from the clicked object, where smaller values indicate higher similarity to the object. This map guides the restoration of the background.
Self-Guided Attention Redirection and Scheduling (SGAR & SGAS):
- SGAR (Self-Guided Attention Redirection): Modulates the self-attention logits ( $S_{self}$ ) during inference. It suppresses object-related key entries using a large negative penalty ( $P_{ob}$ ) and reweights background key entries using a scheduled factor ( $G_{bg}(t)$ ) derived from $M_{bg}$ .
- SGAS (Self-Guided Attention Scheduling): Controls the timing and strength of SGAR to ensure natural background completion. The denoising process is divided into stages:
  - Early stage (first ~20%): No modification.
  - Early-middle stage: SGAR is enabled, with background guidance gradually decaying.
  - Middle stage: Background guidance is disabled, but object suppression remains active.
  - Late stage: All guidance is disabled, allowing the model to complete remaining details autonomously.
Attention Redirection Guidance (ARG):
- To provide controllable output-level guidance, ARG linearly combines the original noise prediction ( $\epsilon(x,t)$ ) and the modulated noise prediction ( $\epsilon'(x,t)$ ).
- The final prediction is calculated as $\epsilon_{ARG} = (1-r) \cdot \epsilon(x,t) + r \cdot \epsilon'(x,t)$ , where $r$ is a user-specified coefficient. A smaller $r$ retains more of the original model's behavior, while a larger $r$ adheres more strictly to the removal guidance.

The system supports both positive clicks (to indicate objects for removal) and negative clicks (to indicate regions for preservation), allowing for complex interactions such as removing occluded objects while preserving foreground elements.

Key Contributions

Interactive Object Removal Tool: The introduction of ClickRemoval, a tool that requires no masks, text descriptions, or additional training. It supports positive and negative click interactions, significantly lowering the usage barrier for non-expert users.
Extensible Attention Redirection Mechanism: A click-driven framework comprising M2N2, SGAR, SGAS, and ARG. This mechanism is not tightly coupled to a specific diffusion backbone and has been successfully implemented across SD1.5, SD2.1, and SDXL1.0.
Complete Open-Source Delivery: The release of a comprehensive software package under the Apache-2.0 license, including source code, configuration files, Docker environments, documentation, demo interfaces, and evaluation scripts.

Experimental Results

The authors evaluated ClickRemoval against strong baselines, including SD-Inpaint, LaMa, PixelHacker, Inpaint Anything, PowerPaint-v2, and AttentiveEraser, using a unified test set derived from Pico-Banana-400K.

Quantitative Metrics: ClickRemoval achieved competitive results without additional training.
- At 1024×1024 resolution (SDXL1.0), it achieved an FID of 8.05 and Local-FID of 15.56, comparable to AttentiveEraser (7.98 / 15.74).
- At 512×512 resolution (SD1.5), it obtained the best overall performance among same-resolution methods with an FID of 9.35, KID of 0.899, and Local-FID of 17.27.
User Study: In a study involving 25 non-expert participants, ClickRemoval ranked first in both resolution settings, receiving 30.48% of votes at 512 resolution and 41.89% at 1024 resolution.
LLM-Assisted Validation: Using a GPT-based multimodal evaluator, ClickRemoval ranked first at 512 resolution (28.85%) and second at 1024 resolution (39.21%), trailing AttentiveEraser by only 1.97 percentage points.
Efficiency: The tool demonstrated acceptable inference overhead, with 1024-resolution runtime comparable to strong baselines and practical efficiency for 512-resolution tasks on consumer hardware (RTX 3090).

Significance

The paper positions ClickRemoval as a significant step toward democratizing high-quality object removal. By removing the dependency on manual masks and text prompts, it offers a "point-and-remove" workflow that is accessible to non-experts while maintaining competitive restoration quality. The authors emphasize that their approach achieves these results without the need for additional training, relying instead on the inherent capabilities of pretrained diffusion models enhanced by their attention redirection framework. The release of the complete open-source package aims to support reproducible research and practical content editing applications.