Technical Summary: Boosting Text Encoder for Personalized Text-to-Image Generation

Problem Statement

Recent advancements in text-to-image (T2I) diffusion models have enabled high-fidelity image synthesis, yet standard prompts often lack the specificity to capture unique concepts or artistic styles. Personalization techniques, such as DreamBooth and Textual Inversion, have emerged to address this by learning specific concepts ($V^\star$) within the token space. However, existing methods suffer from significant computational and storage inefficiencies. Approaches like DreamBooth require fine-tuning extensive portions of the U-Net, leading to substantial storage overhead for each concept. Furthermore, many methods rely on multiple reference images (typically 3–5), limiting their practicality in one-shot scenarios where users possess only a single reference image. Additionally, current encoder-focused adaptations often depend on extensive pre-training on large-scale datasets.

Methodology

The authors propose TextBoost, an efficient one-shot personalization approach that shifts the focus from fine-tuning the image generation module (U-Net) to selectively fine-tuning the text encoder. The methodology is built on three core components:

1. Motivation: The Role of the Text Encoder

Empirical analysis reveals that during the fine-tuning of T2I models, the text encoder undergoes the most significant parameter changes compared to U-Net layers. This suggests that adapting the text encoder is pivotal for personalization. However, naive fine-tuning of the text encoder disrupts its learned semantic space, particularly its auto-regressive causality (where a token's output depends only on itself and preceding tokens).

2. Causality-Preserved Adaptation (CPA)

To fine-tune the text encoder without distorting the semantic integrity of the original sequence, the authors introduce Causality-Preserved Adaptation (CPA).

• Mechanism: CPA attaches lightweight adapters in parallel to specific linear layers within the Transformer blocks of the text encoder.
• Causal Masking: A binary causal mask $m$ is applied to the adapter's output. For a new concept token $V^\star$ located at position $i^\star$, the mask ensures the adapter only influences tokens at positions $i \ge i^\star$.
• Effect: The output embeddings for all tokens preceding $V^\star$ remain identical to the original pretrained encoder, preserving the original contextual understanding while allowing the model to learn the new concept.
• Implementation: Adapters are specifically added to the second feed-forward layer ($fc2$) of the Multi-Layer Perceptron (MLP) blocks within the text encoder, rather than the self-attention projections, based on ablation studies.

3. Extended Textual Embedding

To enhance fine-grained control, the authors propose extending the text embedding space.

• Layer-wise Conditioning: Instead of sharing a single text embedding matrix $H$ across all cross-attention layers of the U-Net, TextBoost generates a set of per-cross-attention embeddings $\{H^{(\ell)}\}$.
• Process: Each embedding is refined by a lightweight adapter specific to that cross-attention layer $\ell$, using the same causality mask.
• Efficiency: Unlike token-space methods (e.g., XTI, NeTI) that require multiple forward passes through the text encoder, this approach performs a single forward pass followed by lightweight adapter transformations, significantly reducing computational overhead.

The method is presented in two variants: TextBoost (using CPA) and TextBoost++ (incorporating both CPA and the extended textual embedding).

Key Contributions

1. Text Encoder Fine-Tuning: The paper presents a novel approach specifically for fine-tuning text encoders for T2I personalization, a direction previously underexplored compared to U-Net adaptation.
2. Causality-Preserved Adaptation (CPA): A strategy that maintains the integrity of the text encoder's original semantic properties and causal relationships during the adaptation of new concept tokens.
3. Extended Text Embedding: A mechanism to extend the embedding space using lightweight adapters, facilitating precise concept adaptation through enhanced cross-attention integration.
4. Efficiency: The method significantly reduces the number of trainable parameters and storage requirements, enabling high-quality personalization suitable for resource-constrained environments.

Experimental Results

The authors evaluated TextBoost on Stable Diffusion v2.1-base and SDXL, comparing it against methods like DreamBooth, Textual Inversion, LoRA, Custom Diffusion, and DisenBooth.

• Quantitative Performance:
  • Fidelity: TextBoost achieves high VQA scores (0.680) and DINOv2 similarity scores (0.560), comparable to or exceeding full fine-tuning methods like DreamBooth (865M parameters) while using only 0.12M parameters.
  • Diversity: The method demonstrates superior generation diversity (0.307) compared to DreamBooth (0.164) and Custom Diffusion (0.136), indicating less overfitting.
  • Comparison with Encoder-based Methods: TextBoost++ outperforms or matches encoder-based methods like ELITE and BLIP-Diffusion, which typically require extensive pre-training, without needing such pre-training.
• Qualitative Results:
  • TextBoost accurately follows complex prompts involving multiple attributes, backgrounds, and styles (e.g., "A V* dog riding a bicycle on the street"), whereas baselines often omit elements or suffer from attribute leakage.
  • In style personalization, TextBoost effectively captures reference styles while maintaining content fidelity, outperforming DisenBooth.
• User Study: A study with 50 participants on Amazon Mechanical Turk showed TextBoost had a higher win-rate than existing methods when users selected images based on subject fidelity and text-image alignment.
• Scalability: The method was successfully applied to SDXL (which uses two text encoders) by fine-tuning only the OpenCLIP ViT-bigG encoder, demonstrating scalability to larger models with minimal parameter updates.

Significance and Claims

The paper claims that TextBoost offers an efficient, scalable, and practically applicable solution for high-quality text-to-image personalization. By demonstrating that the text encoder is a highly effective locus for personalization, the authors argue that high-quality one-shot personalization does not necessarily require extensive modification of the diffusion backbone. Instead, significant gains can be achieved by selectively and structurally adapting the text-side representation space.

The authors position this work as a step toward text-centric approaches for efficient T2I personalization, particularly beneficial in resource-constrained environments (e.g., edge devices, cloud storage with limited capacity) where storage and training efficiency are critical. They suggest that understanding how personalization is encoded and propagated through the conditioning pathway opens new opportunities for designing lightweight, deployable personalization methods.