Technical Summary: CopyCat

Problem Statement

Recent subject-to-image (S2I) models have advanced personalized image generation but struggle to preserve fine-grained subject-specific details. While these models can maintain the coarse identity of a reference subject, they often fail to reproduce subtle distinguishing features. The primary bottleneck is the lack of high-quality, fine-grained identity supervision. Real paired data (reference image and target image of the same subject in different contexts) is prohibitively expensive to collect. Consequently, models rely on synthesized training pairs that effectively teach coarse appearance but fail to capture subtle subject-specific details. Existing solutions that improve data scale require retraining expensive models, while optimization-based personalization methods (e.g., DreamBooth) incur high costs for every new subject.

Methodology

The paper proposes CopyCat, a lightweight model-refinement framework designed to recover missing fine-grained consistency in pretrained S2I models within seconds. The framework consists of two independent components:

1. Fine-grained Consistency LoRA (FCLoRA) Refinement

Instead of collecting new large-scale datasets, CopyCat performs a one-time refinement on an existing pretrained S2I model using a single proxy image.

• Exact Self-Reconstruction Objective: The core innovation is using the proxy image as both the conditioning image ($I_{ref}$) and the reconstruction target ($I_{target}$), such that $I_{ref} = I_{target}$ pixel-wise.
• Mechanism: By aligning the conditioning and target images perfectly, the optimization task eliminates the need to model pose variations, viewpoint changes, or illumination shifts. The objective becomes a pure fine-grained reconstruction problem.
• Efficiency: A lightweight LoRA ($\Delta\theta_{FC}$) with a rank significantly smaller than the original S2I LoRA is attached to the frozen model. It is optimized for only a few steps (approx. 3 seconds on a single GPU) to correct fine-grained visual correspondence errors inherent in the pretrained model.
• Generalization: The refinement does not encode the specific identity of the proxy image. Instead, it learns a transferable capability to preserve fine-grained details, allowing the refined model to be applied to diverse unseen subjects and prompts without further subject-specific optimization.

2. Visual-only Subject-to-Image LoRA

The authors revisit the architecture of S2I models based on double-stream diffusion transformers (e.g., FLUX backbone).

• Observation: Subject-specific knowledge is primarily encoded in visual appearance, while text prompts define the scene or attributes.
• Strategy: The paper proposes removing LoRA adapters from the dedicated text stream of the Double-Stream Blocks while retaining them in the image stream and the subsequent shared Single-Stream Blocks.
• Result: This "Visual-only" strategy reduces trainable parameters and consistently improves subject consistency compared to adapting both streams.

Key Contributions

1. CopyCat Framework: A lightweight, one-time refinement method that significantly improves fine-grained subject consistency in seconds using a single image.
2. Exact Self-Reconstruction Paradigm: The identification that using an identical image for conditioning and reconstruction is the key to effective few-second refinement. This simple objective allows the model to focus exclusively on strengthening fine-grained visual correspondence without modeling semantic variations.
3. Visual-only LoRA Design: A demonstration that adapting only the visual stream in double-stream diffusion transformers outperforms the conventional strategy of adapting both text and visual streams, providing a stronger baseline for S2I training.

Experimental Results

The authors evaluated CopyCat on DreamBench and XVerseBench across single-subject and multi-subject settings using representative models (UNO, UMO, USO, OmniGen, etc.).

• Quantitative Improvements: CopyCat consistently improved identity preservation metrics ($ID_{single}$, $ID_{multi}$, $ID_{INO}$) across all tested backbones while maintaining comparable text alignment ($T_{CLIP}$). For example, on XVerseBench, UNO+CopyCat improved $ID_{single}$ from 59.24 to 69.31 and $IP_{single}$ from 80.75 to 84.95.
• Qualitative Improvements: Visual comparisons show that CopyCat better preserves fine-grained details (e.g., facial features, texture patterns) and reduces identity confusion in multi-subject compositions (e.g., distinguishing between similar animals like cats and dogs).
• Human Face Personalization: Despite being trained without face-specific data, CopyCat substantially improved facial identity consistency, achieving results comparable to specialized models like UMO.
• Ablation Studies:
  • Using a different image of the same subject as the target (rather than the identical image) failed to produce noticeable improvements.
  • Excessive optimization led to overfitting; the optimal refinement was achieved in approximately 3 seconds (4 steps).
  • The Visual-only LoRA strategy consistently outperformed Full LoRA in subject consistency metrics.

Significance and Claims

The paper claims that CopyCat offers a simple and practical direction for improving future subject-to-image personalization models. Its significance lies in:

• Efficiency: It enables the recovery of fine-grained consistency in seconds without requiring additional large-scale training data or subject-specific optimization for every new target.
• Generalizability: The refined model is a "one-time" improvement that generalizes to unseen subjects and prompts, effectively acting as a universal enhancer for existing S2I models.
• Architectural Insight: The Visual-only LoRA finding suggests that text stream adaptation is unnecessary for learning subject-specific visual characteristics, offering a more efficient training paradigm for double-stream diffusion transformers.

The authors emphasize that this approach does not replace the need for high-quality training data in the initial pretraining phase but rather serves as an effective post-training refinement to unlock the underutilized fine-grained consistency capabilities already embedded in pretrained models.