Technical Summary: GenEraser

Problem Statement

Video object removal frequently struggles to simultaneously eliminate target objects and their associated physical effects (e.g., smoke, reflections, light, shadows, and ripples) in out-of-domain scenarios. Existing methods primarily rely on spatial masks, which often fail to capture weakly correlated effects due to complex spatiotemporal ambiguities. For instance, while shadows are strongly correlated with objects, phenomena like light or smoke exhibit weaker correlations, rendering purely mask-guided removal insufficient.

Furthermore, a fundamental optimization conflict exists in current removal models between high-level semantic generalization (successfully removing objects and effects across diverse scenes) and precise pixel-level background preservation (maintaining fidelity in non-edited regions). Single expert models often overfit to local pixels during training, degrading their ability to generalize in open-world scenarios. Additionally, the potential of explicit textual guidance to leverage pre-trained text-to-vision priors remains underexplored, with most methods relying on static prompts or neglecting semantic tokens for physical effects.

Methodology

GenEraser is a novel framework designed to address these challenges through a two-stage training procedure and a decoupled expert architecture.

1. Multi-Conditional Mixture-of-Experts (MC-MoE) with Bipartite Text

To fully exploit the multimodal priors of Diffusion Transformers (specifically the MMDiT architecture), the authors introduce a Multi-Conditional Mixture-of-Experts mechanism.

• Bipartite Text Guidance: Unlike standard prompts, the system utilizes "Bipartite Text" which explicitly describes both the target objects to be erased (in red) and the associated physical effects to be eliminated (in yellow), alongside the desired background state. This allows the cross-attention mechanism to establish direct semantic connections with complex effects.
• Conditional Dropout: During Stage I training, the model is optimized under three distinct branches via random conditional dropout:
  1. Text-only guidance: Forces the model to reconstruct targets using solely textual cues.
  2. Mask-only guidance: Relies on spatial constraints.
  3. Joint (full) guidance: Combines both modalities.
     This strategy cultivates modality-specific experts, enhancing the model's ability to identify complex effects that masks alone might miss.

2. Learnable Deep "CFG" Fusion (LD-CFG)

To address the varying dominance of text versus mask conditions across different scenarios (e.g., text is dominant for removing ripples, while masks are dominant for avoiding the removal of non-target individuals), the authors propose the LD-CFG module.

• Unlike traditional Classifier-Free Guidance (CFG) which relies on manual, heuristic tuning of guidance scales, LD-CFG adaptively balances textual and mask cues.
• In Stage II training, the model processes inputs from text-only, mask-only, and joint branches in parallel. Learnable linear layers within each DiT block fuse these features, allowing the model to dynamically determine the relative dominance of each modality for a specific scene without manual intervention.

3. Decoupled Expert Architecture (Locator and Preserver)

To resolve the trade-off between semantic generalization and pixel alignment, GenEraser employs two distinct expert models:

• Locator (High-Noise Expert): Trained on a diverse mixture of synthetic and real-world data (VOR and ROSE) for fewer optimization steps. Its primary objective is robust semantic generalization to accurately locate and eliminate complex physical effects across open-world scenarios.
• Preserver (Low-Noise Expert): Trained exclusively on "pixel-perfect" synthetic data (ROSE) for a longer duration. Its objective is to ensure high-fidelity reconstruction of non-edited background regions, avoiding the misalignment artifacts often found in real-world datasets.
• Synergy: The Locator handles the removal of objects and effects, while the Preserver ensures the background remains faithful to the original video.

Key Contributions

1. Multi-Conditional Mixture-of-Experts and Bipartite Text: A mechanism that leverages rich multimodal priors to enhance the identification and elimination of complex physical effects by explicitly modeling both objects and their causal effects via text.
2. Learnable Deep "CFG" Fusion Mechanism: A module that achieves continuous, scene-adaptive balance between mask and textual conditions, overcoming the limitations of manual hyperparameter tuning.
3. Decoupled Expert Architecture: A novel design separating the "Locator" and "Preserver" roles to mitigate the inherent conflict between semantic generalization and pixel-level alignment.
4. State-of-the-Art Performance: Extensive experiments demonstrating superior performance across multiple benchmarks while maintaining robust generalization.

Results

GenEraser was evaluated on the ROSE Benchmark, VOR-Eval, and VOR-Wild datasets.

• Quantitative Improvements: The 5B variant of GenEraser achieved a 2.16 dB improvement in PSNR over the previous state-of-the-art (SVOR) on the ROSE Benchmark and a 1.44 dB improvement over EffectErase on VOR-Eval.
• Generalization: On the VOR-Wild benchmark (open-world scenarios), GenEraser achieved an Erasure Preference Rate (EPR) of 0.5872 (5B model), significantly outperforming baselines like ROSE (0.3744) and SVOR (0.4769).
• Qualitative Performance: The model successfully removes a wider range of physical effects—including smoke, deformations, light, mirrors, shadows, and reflections—where competing methods often leave artifacts, background discontinuities, or incomplete erasures.

Significance

The paper claims that GenEraser sets a new state-of-the-art for video object and effect removal. Its significance lies in moving beyond the limitations of purely mask-guided methods by explicitly leveraging text-to-vision priors to handle complex spatiotemporal effects. By decoupling the tasks of semantic erasure and background preservation, the framework successfully resolves the optimization conflict that has historically hindered generalization in open-world scenarios. The introduction of the LD-CFG module further advances the field by enabling adaptive, scene-specific balancing of guidance modalities, eliminating the need for manual, heuristic tuning.