Technical Summary: InsertFuse

Problem Statement

Reference-guided image insertion aims to place a target subject from a reference image into a designated region of a source scene while preserving the subject's appearance and identity, and adapting its scale and spatial placement to the new context. While recent advances in diffusion models and flow matching have improved general image generation and editing, building a unified model capable of handling diverse insertion categories (e.g., accessories, garments, humans, animals, general objects) remains a significant challenge.

Different insertion categories require substantially different generation strategies and modeling objectives. For instance, accessory insertion demands fine-grained texture preservation, virtual try-on requires complex non-rigid deformation, and human insertion prioritizes identity consistency. Directly training a single model on mixed-category data introduces conflicting optimization signals, which can weaken the specialized capabilities required for each task. While recent approaches like Mixture-of-Transformers (MoE) attempt to route inputs to specialized branches, they still rely on joint optimization within a unified training process, leaving category-specific conflicts only partially resolved. Furthermore, existing methods often lack precise spatial control and struggle to balance supervision between the small insertion region and the large background, leading to background-dominated optimization or scale-dependent performance issues.

Methodology: InsertFuse

InsertFuse proposes a unified framework that decouples category-specific expertise learning from cross-category capability consolidation. The framework operates in two distinct stages and introduces three key technical components to enhance spatial control and reference fidelity.

1. Two-Stage Training Framework

• Stage I: Category-Specific Expert Learning. The framework initializes multiple experts from a pretrained flow-matching image-editing model (specifically Qwen-Image-Edit-2511). Each expert is trained independently on a specific insertion category (accessory, animal, garment, general-object, human). This allows each expert to acquire specialized knowledge from its corresponding data distribution without interference from other categories.
• Stage II: Cross-Category Consolidation via IOPD. A unified "student" model is initialized from the same pretrained backbone. All category-specific experts are frozen. The student is trained using Insertion On-Policy Distillation (IOPD). In this process, the student generates its own trajectories (rollouts). For each state visited by the student, the matched frozen expert (selected based on the category label) provides a velocity target. This distillation transfers the specialized insertion behaviors and reference fidelity of the experts into the single student model, mitigating the interference caused by direct joint training.

2. Key Technical Components

To equip the experts and the student with precise control, InsertFuse introduces three novel modules:

• Token-Aligned Geometry Conditioning (TAGC):
  Instead of simply concatenating a binary mask with visual inputs (which requires the model to implicitly learn mask-to-token correspondence), TAGC explicitly maps the insertion mask to the visual token grid. It constructs a compact geometric representation $G(M)$ containing the mask, a signed distance transform, and a boundary proximity map. A lightweight geometry encoder extracts features that are projected and injected residually into both the target and source-context latent branches. This establishes explicit spatial correspondence without adding extra image-like mask inputs.

• Region-Balanced Flow Matching (RBFM):
  Standard flow-matching objectives uniformly average prediction errors, which can be dominated by the large background region, leaving small insertion subjects under-supervised. RBFM addresses this by separately normalizing prediction errors inside and outside the insertion region. It defines separate loss terms for the insertion region ($\ell_{ins}$) and the background ($\ell_{bg}$), each normalized by their respective effective areas. This prevents the supervision strength from scaling with the insertion size and ensures balanced optimization for both the subject and the background.

• Reference Classifier-Free Guidance (Reference CFG):
  To strengthen the preservation of the reference subject's appearance and identity, the framework introduces Reference CFG. During expert training, the reference condition is randomly dropped with probability $p_{drop}$. At inference, the model computes the difference between the prediction with the reference and the prediction without it (while keeping scene and geometry conditions fixed). This difference isolates the reference-induced velocity direction, which is then amplified to guide the generation. This enhanced supervision is transferred to the student via IOPD, allowing the unified model to maintain strong reference consistency without needing an additional CFG branch during inference.

Key Contributions

1. Unified Framework with Decoupled Learning: The proposal of InsertFuse, a framework that separates the learning of category-specific expertise from cross-category consolidation. It utilizes Insertion On-Policy Distillation (IOPD) to transfer specialized policies into a single student, effectively mitigating the interference inherent in direct joint training.
2. Enhanced Spatial and Regional Control: The introduction of Token-Aligned Geometry Conditioning (TAGC) for explicit mask-to-token alignment and Region-Balanced Flow Matching (RBFM) to normalize supervision errors, ensuring stable performance across objects of varying scales and preventing background dominance.
3. Strengthened Reference Fidelity: The proposal of Reference CFG, which isolates and amplifies visual reference guidance under fixed scene conditions, improving the preservation of appearance and identity.
4. State-of-the-Art Performance: Extensive experiments demonstrating that InsertFuse outperforms existing reference-guided insertion methods and unified image editing models across multiple metrics.

Experimental Results

The authors evaluated InsertFuse on the public AnyInsertion benchmark and a custom multi-category test set covering accessories, animals, garments, general objects, and humans.

• Quantitative Performance: InsertFuse achieved state-of-the-art results on most metrics. On the AnyInsertion benchmark, it scored 0.7838 (DINO-I) and 0.9156 (CLIP-I), indicating strong reference fidelity. On the multi-category test set, it achieved 0.7148 (DINO-I) and 0.9020 (CLIP-I). It also demonstrated superior reconstruction quality with high PSNR (24.26/24.40) and SSIM, and low LPIPS and FID scores compared to baselines like AnyDoor, Insert Anything, A2-Edit, and general-purpose models like FLUX.1 Kontext and Qwen-Image-Edit.
• Qualitative Performance: Visual comparisons show that InsertFuse better preserves reference-specific details (e.g., textures, logos, patterns) and adapts more naturally to target geometry, pose, and occlusion relationships compared to competing methods, which often suffer from structural artifacts or identity drift.
• User Study: In a multi-way user study, InsertFuse received the highest preference shares (51.8% for Reference Fidelity, 58.2% for Insertion Quality, and 55.4% for Overall Quality) across all criteria.
• Ablation Studies:
  • TAGC significantly improved geometric alignment and boundary transitions compared to baseline mask concatenation or raw injection.
  • RBFM provided consistent gains over standard global flow matching, particularly for small insertion regions, by preventing background dominance.
  • Reference CFG improved reference-oriented metrics and preserved fine-grained textures.
  • IOPD was shown to effectively consolidate expert capabilities, significantly outperforming direct joint training (which suffered from cross-category gradient interference, as evidenced by negative gradient cosine similarities between certain category pairs like human-garment).

Significance

The paper claims that InsertFuse provides a unified and practical solution for diverse reference-guided image insertion tasks. By decoupling the acquisition of specialized expertise from the consolidation of cross-category capabilities, the framework addresses the fundamental conflict in training unified models for heterogeneous tasks. The proposed components (TAGC, RBFM, Reference CFG) offer explicit geometric guidance and balanced regional optimization, leading to strong performance in reference preservation, spatial adaptation, and generation quality. The results suggest that separating specialization from consolidation is a more effective strategy than attempting to resolve category conflicts solely through joint optimization or routing mechanisms.