💻 computer science

PANDORA: Pixel-wise Attention Dissolution and Latent Guidance for Zero-Shot Object Removal

PANDORA is a novel zero-shot object removal framework that leverages Pixel-wise Attention Dissolution and Localized Attentional Disentanglement Guidance to erase objects from natural images using pre-trained diffusion models without requiring fine-tuning, prompts, or optimization, thereby achieving superior visual fidelity and semantic plausibility compared to existing methods.

Original authors: Dinh-Khoi Vo, Van-Loc Nguyen, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dinh-Khoi Vo, Van-Loc Nguyen, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful photograph of a park, but there's an ugly trash can sitting right in the middle of a flower bed. You want to delete the trash can and have the flowers and grass fill in that space naturally, as if the trash can was never there.

Doing this with old computer programs was like trying to fix a hole in a wall by pasting a piece of wallpaper from a different room onto it. It often looked fake, blurry, or mismatched.

Newer AI programs (called "Diffusion Models") are like master painters who can imagine what a scene should look like. But even these master painters have a problem: if you tell them to "remove the trash can," they often get confused. They might try to paint a new trash can, or they might leave a weird ghostly outline of the old one because they are too attached to the original image.

PANDORA is a new tool that fixes this. It's like giving the master painter a set of special instructions that don't require them to go to art school (fine-tuning) or read a long, complicated poem (prompts) to understand what to do. It works instantly, right out of the box.

Here is how PANDORA works, using three simple metaphors:

1. The "Memory Guard" (Background Preservation)

Imagine the AI is looking at the whole photo. It needs to know which parts to keep exactly as they are (the flowers, the sky) and which parts to change (the trash can).

What PANDORA does: It puts a "Do Not Touch" sign on the parts of the image you want to keep. It locks the background in place so the AI doesn't accidentally repaint the sky or blur the trees while it's working on the trash can.

2. The "Social Distancing" Rule (Pixel-wise Attention Dissolution)

This is the magic trick. Inside the AI's brain, every pixel (dot of color) talks to every other pixel. The pixels that make up the trash can are "talking" loudly to each other, saying, "We are a trash can! We belong together!"

The Problem: If the AI just tries to erase the trash can, those pixels keep whispering to each other, making the trash can reappear or look weird.
What PANDORA does: It acts like a strict bouncer at a club. It looks at the pixels inside the trash can area and says, "Stop talking to your friends who make you look like a trash can!" It cuts the connection between those pixels and the rest of the image.
The Result: Once the trash can pixels are "socially isolated," they can't hold their shape anymore. They dissolve. The AI then looks at the surrounding flowers and grass and says, "Okay, since the trash can is gone, let's just fill this space with more flowers."

3. The "Steering Wheel" (Localized Attentional Disentanglement Guidance)

Even after cutting the connections, the AI might still drift back toward painting a trash can because that's what it saw in the original photo. It's like driving a car that keeps trying to turn back to the original route.

What PANDORA does: It grabs the steering wheel and gently pushes the AI's "imagination" away from the idea of a trash can and toward the idea of an empty, clean park. It guides the AI's thought process so it stays on the path of "clean background" and doesn't accidentally wander back into "trash can territory."

Why is this a big deal?

No Training Needed: You don't have to teach the AI anything. It already knows how to paint; PANDORA just gives it the right rules for this specific job.
No Magic Words: You don't have to type "remove trash can, make it look realistic." It just works based on the shape you draw.
Handles Crowds: If you have a photo with ten people standing close together and you want to remove all of them at once, older tools get confused. PANDORA can remove a whole crowd in one go without making the background look messy.

In short: PANDORA is a smart, zero-maintenance tool that tells an AI painter exactly how to "un-paint" an object by cutting its mental connections and guiding its imagination, leaving you with a perfect photo where the object never existed.

1. Problem Statement

Object removal in natural images is a challenging task in semantic image editing. The goal is to selectively erase undesired regions and reconstruct the void with content that blends naturally with the scene's semantics and visual flow.

Limitations of Existing Methods:
- Patch-based/GANs: Often result in texture inconsistencies, unnatural blending, and rigid artifacts.
- Fine-tuned Diffusion Models (e.g., Stable Diffusion Inpainting): While realistic, they require extensive fine-tuning or prompt engineering, which is impractical for low-resource settings and limits scalability. They often struggle with incomplete removal or introduce artifacts.
- Zero-Shot Methods: Existing approaches (e.g., CPAM, Attentive Eraser) often fail to handle multi-object removal, leaving residual objects due to weak foreground-background disentanglement or requiring sensitive manual tuning.

2. Methodology: PANDORA Framework

PANDORA is a zero-shot framework that operates directly on pre-trained text-to-image diffusion models (specifically Stable Diffusion) in a single forward pass. It requires no fine-tuning, no text prompts, and no inference-time optimization.

The pipeline consists of three main stages:

A. Background Preservation Adaptation (BPA)

To ensure the non-masked background remains unaffected, PANDORA utilizes a preservation module adapted from CPAM.

Process: The input image is inverted into the latent noise space using DDIM inversion. Intermediate latent states are stored.
Mechanism: During the denoising process, the query features ( $Q$ ) from the current noise are paired with key ( $K_i$ ) and value ( $V_i$ ) features extracted from the stored latent noise of the original image.
Result: This restricts attention for background queries to background regions, preserving structural and visual fidelity outside the mask.

B. Pixel-wise Attention Dissolution (PAD)

This is the core mechanism for erasing the object by disrupting its semantic connections within the self-attention layers.

Concept: Instead of simply masking out regions, PAD actively "dissolves" the object's information by nullifying the strongest correlations in the self-attention flow.
Mechanism:
1. Compute attention logits ( $S$ ) between query features ( $Q$ ) and stored keys ( $K_i$ ).
2. Apply percentile-based thresholding: Identify the top- $k$ most correlated keys for each masked query pixel.
3. Dissolution: Set the attention scores for these top- $k$ connections (and the mask itself) to $-\infty$ .
4. Recombination: The final output blends the dissolved object attention with the preserved background attention.
Effect: This disconnects masked query pixels from dominant foreground regions, forcing the model to reconstruct the masked area using only the surrounding background context, effectively erasing the object.

C. Localized Attentional Disentanglement Guidance (LADG)

This module refines the denoising trajectory in the latent space to ensure clean removal and suppress residual artifacts.

Concept: A spatially gated guidance mechanism that pushes the latent distribution away from the original (unconditioned) trajectory specifically inside the object mask.
Mechanism:
- It calculates a weighted noise prediction: Outside the mask, it preserves the conditional trajectory (maintaining scene fidelity). Inside the mask, it steers the latent noise toward a conditional prediction that is distinct from the unconditioned object trajectory.
- Formula: $\hat{\epsilon} = (1-M) \odot \epsilon_c + M \odot [\alpha_t \epsilon_c + (1-\alpha_t) \epsilon_u]$ , where $\alpha_t$ controls the degree of steering.
Effect: Accelerates object removal and mitigates residual artifacts, producing smoother, cleaner results.

3. Key Contributions

PANDORA Framework: A novel zero-shot object removal method that leverages pre-trained diffusion models without fine-tuning, prompts, or optimization.
Pixel-wise Attention Dissolution (PAD): A technique that dissolves object information by disconnecting masked query pixels from their most correlated keys in self-attention, enabling fine-grained, non-rigid removal.
Localized Attentional Disentanglement Guidance (LADG): A guidance strategy that steers latent noise away from object regions while preserving the background, refining the reconstruction to eliminate artifacts.
New Benchmark Dataset: The authors constructed a challenging dataset comprising 75 single-object, 17 multi-object, and 94 mass-similar object samples (e.g., clusters of fruits or birds) to evaluate diverse removal scenarios.

4. Experimental Results

The authors evaluated PANDORA against state-of-the-art (SOTA) fine-tuned methods (LaMa, PowerPaint, SD2-Inpaint) and zero-shot methods (CPAM, Attentive Eraser).

Quantitative Performance:
- PANDORA achieved the highest CLIP scores (indicating effective semantic removal of objects) among zero-shot methods.
- It maintained competitive FID, LPIPS, and MSE scores, demonstrating high background realism.
- Ablation Studies: Removing PAD caused a significant performance drop (incomplete erasure), while removing LADG slightly reduced accuracy (residual artifacts).
Qualitative Performance:
- PANDORA successfully handled single objects, multiple distinct objects, and densely packed "mass-similar" objects (e.g., a flock of birds) in a single pass.
- Unlike competitors, it avoided residual artifacts and texture inconsistencies, even in complex scenes.
User Study: In a study with 20 participants, PANDORA received a 47.5% User Preference Rate (UPR), significantly outperforming all other methods (the next highest was Attentive Eraser at 22%).

5. Significance

Accessibility: By eliminating the need for fine-tuning or prompt engineering, PANDORA makes high-quality object removal accessible for low-resource environments and broad research applications.
Scalability: It is the first zero-shot method capable of robustly handling multi-object and mass-similar object removal, a task where previous methods often failed.
Efficiency: The single-pass inference makes it computationally efficient compared to iterative optimization methods.
Generalization: The method demonstrates strong generalization across different Stable Diffusion backbones (v1.5 and v2.1).

Limitations: The method relies on accurate binary masks; imprecise segmentation can degrade results. The fixed percentile threshold for attention dissolution may occasionally over- or under-filter, though empirical results suggest a 2–5% suppression range is optimal. Future work aims to introduce adaptive thresholds and automatic region selection.