Technical Summary: EraseLoRA

Problem Statement

Object removal in images requires not only eliminating a masked target but also faithfully reconstructing the occluded background with structural and contextual fidelity. While recent dataset-free approaches utilizing diffusion models attempt to achieve this by manipulating self-attention mechanisms to prevent the model from referencing the masked region, the authors identify two critical failure modes in these methods:

1. Misinterpretation of Foreground: Existing methods treat the entire unmasked region as background. Consequently, non-target foreground objects (e.g., a chair next to a removed table) are misinterpreted as background and inadvertently regenerated within the masked area.
2. Uniform Attention Constraints: These methods apply uniform attention blocking or redirection without distinguishing between diverse background subtypes (e.g., grass, pavement, sky). This leads to textural blurring, structural misalignment, and the failure to coherently integrate multiple background cues.

The authors argue that these failures stem from a lack of explicit background-aware reasoning, where the model fails to distinguish between what should be removed, what should be preserved as foreground, and what constitutes the clean background to be reconstructed.

Methodology

The paper proposes EraseLoRA, a dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. The method operates in two distinct stages:

Stage 1: Background-aware Foreground Exclusion (BFE)

This stage leverages a Multimodal Large Language Model (MLLM) to perform semantic reasoning on a single image-mask pair.

• Semantic Partitioning: The MLLM analyzes the image to identify and classify three distinct categories: the target foreground (to be removed), non-target foregrounds (other objects in the scene that must be preserved), and background subtypes (the occluded content to be reconstructed).
• Mask Generation: Using the MLLM's output tags, the system employs Tag2Mask models (e.g., Grounding DINO and SAM2) to generate precise binary masks for the target ($M_T$), non-target foregrounds ($M_F$), and clean background ($M_B$).
• Exclusion: By explicitly isolating non-target foregrounds, the method prevents these distractors from being used as reference cues for reconstruction, a key differentiator from prior work that treats all unmasked pixels as background.

Stage 2: Background-aware Reconstruction with Subtype Aggregation (BRSA)

This stage performs test-time optimization using Low-Rank Adaptation (LoRA) to reconstruct the masked region without explicit attention blocking.

• Optimization Objective: The method jointly optimizes two complementary losses to guide the diffusion model:
  1. Background Reconstruction Loss ($L_{recon}$): Enforces fidelity to the original image by minimizing the difference between the reconstructed latent and the input latent specifically within the clean background mask ($M_B$). This anchors the generation to the true background structure.
  2. Background Puzzle Loss ($L_{puzzle}$): Treats inferred background subtypes as distinct "puzzle pieces." It maximizes the overlap between the dominant attention map and the valid regions (target mask + clean background), ensuring that attention is concentrated on relevant background cues rather than non-target foregrounds.
• Process: The LoRA adapters are inserted into the diffusion backbone and optimized for a fixed number of iterations (default 500) using the combined loss ($L_{total} = L_{recon} + \lambda L_{puzzle}$). The backbone parameters remain frozen.

Key Contributions

The paper outlines four primary contributions:

1. Identification of Failure Modes: The authors formally identify that the unintended regeneration of non-target foregrounds is a fundamental failure mode in current dataset-free object removal methods, caused by a lack of background-aware reasoning.
2. EraseLoRA Framework: A novel, model-agnostic, dataset-free framework that combines MLLM-guided separation of foregrounds and backgrounds with a multi-background-aware test-time adaptation scheme. It avoids explicit attention manipulation, thereby preserving fine textures.
3. New Annotations and Metrics: The authors provide three-label ground-truth annotations (target, non-target foreground, background) for existing benchmarks and introduce evaluation metrics (Foreground Similarity and Background Similarity) specifically designed for unpaired object removal settings.
4. Performance Gains: The method demonstrates significant improvements over state-of-the-art dataset-free methods, achieving at least a 23% increase in background fidelity and nearly halving unwanted foreground re-generation, while outperforming dataset-driven approaches despite requiring no training data.

Experimental Results

The authors evaluated EraseLoRA on two benchmarks: OpenImages V7 and RORD, using various diffusion backbones (SD3.5-M, SDXL, FLUX.1).

• Quantitative Performance: On OpenImages V7, EraseLoRA improved Background Similarity (BG Sim.) from 0.605 (baseline) to 0.743 and reduced Foreground Similarity (FG Sim., where lower is better) from 0.286 to 0.151. Similar gains were observed on RORD.
• Comparison: EraseLoRA outperformed all compared dataset-free methods (e.g., AttentiveEraser, DesignEdit) and surpassed dataset-driven methods (e.g., PowerPaint, SmartEraser) in both background fidelity and foreground suppression.
• Ablation Studies:
  • BFE Efficacy: Plugging the BFE module into prior methods improved their BG Sim. by up to 6.6% and reduced FG Sim. by 8.6%, confirming that foreground exclusion is a critical factor.
  • Loss Components: The combination of $L_{recon}$ and $L_{puzzle}$ was shown to be necessary; using either alone resulted in either faint foreground traces or structurally inconsistent completions.
  • Flexibility: The method demonstrated robustness across different MLLMs (from 7B to 78B parameters) and Tag2Mask models, with larger MLLMs providing slightly better reasoning but lightweight models still yielding substantial gains.

Significance and Claims

The paper positions EraseLoRA as a significant step forward in dataset-free object removal. Its primary significance lies in shifting the paradigm from "attention surgery" (which often degrades image quality) to "background-aware reasoning." By utilizing the reasoning capabilities of MLLMs to understand scene semantics and the generative fidelity of diffusion models enhanced by test-time adaptation, EraseLoRA achieves:

• Faithful Restoration: It reconstructs backgrounds that are structurally and texturally consistent with the original scene, avoiding the hallucination of new objects.
• Generalizability: As a plug-and-play module, it is applicable to diverse diffusion backbones without requiring retraining or paired datasets.
• Efficiency vs. Quality Trade-off: While it incurs computational overhead due to MLLM inference and test-time optimization, the authors claim this is a justifiable trade-off for the substantial gains in removal quality and the elimination of unwanted foreground regeneration. They also note that efficiency can be improved via lightweight MLLMs and early stopping strategies.

The authors conclude that EraseLoRA establishes an extensible formulation for dataset-free object removal, effectively addressing the limitations of previous approaches by explicitly reasoning about what lies behind the target and what should be excluded from the reconstruction process.