When Few Steps Are Enough: Training-Free Acceleration of Identity-Preserved Generation

Technical Summary: Training-Free Acceleration of Identity-Preserved Generation

Problem Statement
Identity-preserved image generation, essential for personalized content creation and avatar generation, typically relies on multi-step diffusion backbones (e.g., 28 steps). While adapter-based methods like InfiniteYou successfully inject identity embeddings (e.g., ArcFace) into models like FLUX, the deployment cost remains high due to the standard many-step sampling schedules inherited from the base backbone. The implicit assumption in current pipelines is that additional denoising steps are necessary to improve all objectives simultaneously: visual quality, text alignment, and identity fidelity. This paper challenges that assumption, asking whether identity-conditioned generators require the full trajectory to preserve identity.

Methodology
The authors propose a training-free distilled backbone replacement strategy. The core method involves:

1. Backbone Substitution: Replacing the standard 28-step FLUX.1-dev backbone with the distilled 4-step FLUX.1-schnell backbone.
2. Configuration Changes: Only two configuration adjustments are required: changing the base model path and disabling classifier-free guidance (CFG), which is standard for FLUX.1-schnell.
3. Frozen Components: The identity adapter (InfuseNet), the face encoder (ArcFace), and all conditioning scales remain completely frozen. No weights are retrained, fine-tuned, or LoRA-adapted.

The hypothesis driving this approach is that identity fidelity enters an "early effective regime" within the first few denoising steps (observed around 4–8 steps), where the coarse facial structure and identity-specific geometry are formed. Subsequent steps primarily refine visual details, sharpness, and contrast, offering diminishing returns for identity preservation.

Key Contributions

• Training-Free Transfer: Demonstrates that a frozen InfuseNet identity adapter trained on FLUX.1-dev transfers directly to the distilled FLUX.1-schnell backbone without retraining.
• Improved Efficiency–Fidelity Tradeoff: Achieves a 5.9× reduction in latency while simultaneously improving identity similarity and perceptual distance compared to the 28-step baseline.
• Identification of an Early Identity-Effective Window: Establishes that identity similarity saturates early in the denoising trajectory, making the full 28-step trajectory inefficient for identity preservation.
• Mechanistic Analysis: Through adapter ablations and attention-stream norm probes, the paper shows that identity formation depends heavily on the adapter in early steps, while the relative contribution of the conditioning stream decreases as sampling proceeds. Later steps focus on visual refinement rather than identity formation.
• Broader Sanity Checks: Preliminary sweeps on SDXL (style adapter) and SD1.5 (object adapter) suggest similar diminishing returns after intermediate steps, indicating this phenomenon may extend beyond FLUX.

Experimental Results
Evaluated on 28 FFHQ identities across three prompts (portrait, beach, professional suit):

• Latency: Reduced from 10.29s to 1.73s per image (5.9× speedup).
• Identity Similarity (ArcFace): Improved from 0.5872 to 0.6150 (+0.0278).
• Perceptual Distance (LPIPS): Improved from 0.7253 to 0.7097 (−0.0156).
• Step Sweep Analysis: In FLUX.1-dev sweeps, identity similarity rose rapidly, reaching a high-fidelity regime by 4–8 steps (e.g., 0.615 at 8 steps) before saturating or slightly declining. In contrast, visual metrics like sharpness and face detection confidence continued to improve significantly through step 28.
• Adapter Necessity: Ablation studies confirmed that without the adapter (or with a weak scale), identity similarity remained near zero, proving the adapter drives identity formation, which occurs early in the process.

Significance and Claims
The paper positions distilled backbone replacement as a simple, deployment-oriented strategy to improve the efficiency–fidelity tradeoff in identity-preserved generation. The authors explicitly state that 4 steps are not a universal optimum; rather, the optimal step count is subject- and prompt-dependent. However, the findings identify a robust "early identity-effective window" where identity is already established.

The significance lies in decoupling the sampling schedule from the assumption that more steps always yield better identity preservation. By recognizing that identity formation saturates early, the authors demonstrate that existing distilled backbones can be leveraged as "drop-in" replacements to drastically reduce inference costs without sacrificing, and in this case, even improving, identity fidelity. The work suggests that sampling budget should be treated as a primary deployment parameter for adapter-conditioned generation.

Limitations Acknowledged
The authors note that the optimal step count varies by subject and prompt, and that ArcFace similarity is an imperfect metric, particularly under strong stylization. The mechanistic probes are diagnostic rather than definitive causal proof. Furthermore, the strongest results rely on the specific compatibility between FLUX.1-dev and FLUX.1-schnell; generalization to other backbone families is not guaranteed without further study.