💻 computer science

AdaEraser: Training-Free Object Removal via Adaptive Attention Suppression

AdaEraser is a training-free object removal framework that outperforms existing methods by employing a token-wise adaptive attention suppression strategy, which dynamically modulates self-attention based on estimated object presence to balance effective removal with high-quality background reconstruction.

Original authors: Dingming Liu

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dingming Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photo of a beautiful park, but there's an ugly trash can right in the middle of the frame. You want to delete the trash can and have the grass and trees fill in that empty space naturally, as if the trash can was never there. This is called "object removal."

For a long time, computers struggled with this. If they just "erased" the trash can, the background often looked blurry, warped, or like a weird patch of static.

Enter AdaEraser, a new tool that fixes this problem without needing to be "taught" with thousands of new photos. Here is how it works, explained simply:

The Problem: The "Brute Force" Mistake

Previous methods tried to remove objects by simply blocking the computer's "eyes" from looking at the trash can area. Imagine you are trying to paint over a spot on a wall. If you put a blindfold on the painter and tell them, "Don't look at this spot, just paint whatever you think belongs there," they might guess wrong. They might paint a patch of sky where a tree should be, or a solid wall where a window belongs.

In technical terms, these old methods aggressively suppressed the computer's internal "attention" (its focus) on the object. While this successfully made the object disappear, it also stopped the computer from using the surrounding context (the trees, the grass, the light) to figure out what should be in the empty space. The result was often a messy, low-quality background.

The Solution: The "Smart Dimmer Switch"

AdaEraser is like giving the painter a smart dimmer switch instead of a blindfold.

Instead of completely blocking the computer's view of the object area, AdaEraser watches the object disappear in real-time and adjusts the "volume" of the computer's attention accordingly.

The "Presence Score" (The Smoke Detector):
As the computer tries to erase the object, it constantly checks: "Is the object still there?"
It does this by comparing the "thoughts" (attention maps) of the image before the erasing started with the "thoughts" during the erasing process.
- Early in the process: The object is still clearly visible in the computer's "mind." The system says, "Okay, the object is still here. We need to suppress it heavily."
- Later in the process: The object is fading away, and the background is starting to form. The system says, "Ah, the object is mostly gone. We can stop suppressing so hard and let the background paint itself naturally."
Token-by-Token Precision (The Surgeon vs. The Sledgehammer):
Old methods treated the whole object area as one big block. AdaEraser is much more precise. It looks at the image as millions of tiny dots (called "tokens").
- Some dots might be the trash can's handle; others might be the shadow it casts.
- AdaEraser checks each dot individually. If a specific dot has already "forgotten" it was part of the trash can and is now thinking about the grass, the system stops suppressing that dot immediately. This allows the grass to grow back naturally, pixel by pixel.

The Result

By using this adaptive approach (changing the suppression strength dynamically), AdaEraser achieves two things at once:

It successfully removes the unwanted object.
It preserves the computer's ability to "imagine" the correct background, because it doesn't shut down the computer's creative brain too early.

Why It's Special

No Training Required: Most advanced tools need to be trained on massive datasets of "before and after" photos. AdaEraser works "out of the box" on existing models. It's like having a tool that works on any car without needing a mechanic to tune it first.
Better Than Trained Models: Surprisingly, this "no-training" method actually produces better results than many methods that did require expensive training.

When It Might Struggle

The paper notes that if the background is very confusing (like a wall with a repeating pattern that looks exactly like the object), or if the object has complex shadows that the user didn't include in the "erase" mask, the tool might leave a small trace or make a slight error. But for most standard photos, it creates a seamless, high-quality edit.

In short: AdaEraser is a smart eraser that knows exactly when to stop erasing and start painting, ensuring the background looks natural and the object is gone.

Technical Summary: AdaEraser

Problem Statement
Object removal in images requires not only the precise elimination of a user-specified target but also the plausible reconstruction of the background in the vacated regions. While recent training-free methods utilizing pre-trained diffusion models have shown promise, they often face a fundamental conflict: effectively removing an object versus generating high-quality background content. Existing approaches, such as AttentiveEraser, typically employ indiscriminate suppression of self-attention within the masked object regions. While this successfully removes the target, it inadvertently disrupts the background generation process. Because background reconstruction relies on the generative capabilities of the pre-trained model—which assumes unmodulated self-attention—blocking attention entirely in these regions leads to structural artifacts, visual distortions, and low-quality background synthesis.

Methodology: AdaEraser
To resolve this conflict, the authors propose AdaEraser, a training-free framework that dynamically modulates attention suppression based on the estimated presence of the target object concept during the denoising process. The core insight is that object removal and background generation are competing goals that require a balanced, adaptive approach rather than a static, binary suppression.

The method operates through three primary steps:

Reference and Target Attention Extraction:
- Reference: For a source image $I_{src}$ , the model generates reference self-attention maps ($SAsrc$) by applying a "t steps diffusion + 1 step denoising" scheme. This preserves the original image's feature information under noisy conditions without requiring a full inversion process.
- Target: Simultaneously, the model performs the standard object removal denoising process on a target latent initialized from the source image, generating target self-attention maps ($SAtgt$).
Token-Wise Presence Score Estimation:
- The authors observe that self-attention maps encode contextual relationships and semantic features. They hypothesize that the similarity between the reference map (containing the object) and the target map (where the object is being removed) correlates with the object's residual presence.
- A token-wise presence score $p(i)$ is computed for each token $i$ within the object mask using cosine similarity between $SAtgt(i)$ and $SAsrc(i)$.
- As the denoising process progresses and the object is removed, the semantic content of the target region diverges from the reference, causing the similarity score to decrease. This score serves as a heuristic proxy for the object's presence level.
Adaptive Attention Suppression:
- Instead of fixed suppression, AdaEraser calculates a suppression coefficient $\eta(i)$ for each token based on its presence score: $\eta(i) = 1 - p(i)$ for tokens in the mask, and $1$ otherwise.
- This coefficient modulates the self-attention scores in the denoising UNet. When the object is clearly present (high similarity), suppression is strong. As the object is removed and the background begins to form (low similarity), suppression is relaxed, allowing the model to utilize its full generative capacity to reconstruct the background naturally.
- Finally, the foreground (denoised target) and background (original source) are blended using the mask at each timestep to ensure consistency in non-removed regions.

Key Contributions

Adaptive Regulation Strategy: The paper identifies that high-quality object removal requires adaptive regulation of attention suppression based on the residual presence of the object, rather than static suppression.
Token-Wise Heuristic: It proposes using token-wise self-attention map similarity as an effective, training-free proxy to estimate object presence in the latent space, overcoming the difficulty of direct object detection in noisy latents.
AdaEraser Framework: The authors introduce a complete training-free framework that integrates this adaptive strategy, achieving a balance between object elimination and background restoration.

Experimental Results
Extensive experiments were conducted on the Mulan and OABench datasets using Stable Diffusion XL (SDXL) as the backbone, with additional validation on SD1.5, SD2.1, and FLUX.

Quantitative Performance: AdaEraser outperformed both training-based methods (e.g., PowerPaint, RORem, SmartEraser) and state-of-the-art training-free methods (e.g., AttentiveEraser, OmniPaint) across all metrics, including FID, LPIPS, PSNR, ReMOVE, and CFD. Notably, it achieved the highest Average Human Ranking (AHR) in user studies.
Qualitative Performance: Visual comparisons demonstrated that AdaEraser effectively removes objects while preserving scene context and avoiding the structural distortions and artifacts common in methods that use full attention suppression.
Ablation Studies: The authors validated that token-wise adaptive suppression significantly outperforms timestep-based or region-based suppression strategies. They also confirmed that using a dynamic reference ( $x_{src}^t$ ) aligned with the current noise level yields superior results compared to fixed-noise references.

Significance and Claims
The paper claims that AdaEraser provides a new perspective on object removal by demonstrating that training-free methods can surpass training-based approaches when the generative prior is leveraged more intelligently. By dynamically adjusting suppression strength, AdaEraser avoids the drawbacks of indiscriminate attention suppression, thereby preserving the pre-trained model's generative capability for background reconstruction. The authors position this work as a significant step toward high-quality, data-efficient image editing that does not require the costly acquisition of specialized datasets or model fine-tuning.

Limitations and Future Directions
The paper acknowledges that performance relies on mask quality; while robust to slightly loose masks, it struggles with incomplete masks that fail to cover object-induced side effects (e.g., shadows). Additionally, the method may face challenges in scenes with highly similar textures or semantically ambiguous content where the generative prior alone is insufficient for complex reconstruction. The authors suggest that future work could involve incorporating stronger structural guidance or automatically identifying side effects to improve robustness. They also note that the method is less compatible with highly accelerated distilled models (e.g., SDXL-Lightning) due to the reduced informativeness of attention dynamics in fewer denoising steps.