Technical Summary: Style-CCL

Problem Statement

Content-preserving style transfer using Diffusion Transformers (DiTs) faces significant challenges due to the entanglement of content and style features. Current models suffer from two primary issues:

1. Leakage/Invasion: Characteristics from the style reference (subject, background, or facial identity) are over-transferred, polluting the content reference.
2. Learning Contradictions: When training on mixed style categories in a single stage, semantic-related style transformations (e.g., 2D/3D cartoons, vector designs) hinder the learning of texture-related style transformations (e.g., oil painting, dense line drawings). This one-stage paradigm also fails to preserve content integrity, particularly when mixing clean collected data with synthetic data.

Methodology

1. Architecture: SC-DiT

The authors propose a Style-Content DiT (SC-DiT) built upon the FLUX-dev foundation. The architecture decouples style and content processing through:

• Dual Branches: Separate branches for style and content conditions, utilizing VAE encoders to extract visual features.
• Decoupled Embeddings: The model employs separate Rotary Position Embeddings (RoPE) for style and content branches.
• Causal Masking: A causal attention mechanism is applied to forbid interactions between style and content queries, as well as between style/content and text/noise branches, ensuring feature disentanglement.
• LoRA Injection: Style and content conditions are injected via separate Low-Rank Adaptation (LoRA) modules attached to the query matrices.

2. Dataset Construction

To address the scarcity of real-world style triplets, the authors constructed a million-scale training set:

• Clean Data ($D_{pure}$): Collected and purified [Style Ref, Content Ref, Target] triplets from existing datasets and open-source LoRAs (approx. 330k triplets, 30 styles).
• Synthetic Data ($D_{synth}$): A reverse triplet synthesis pipeline was developed to generate triplets from "in-the-wild" style images. An image editing model converts stylized images into photorealistic targets to create synthetic content-style pairs (approx. 1 million triplets).

3. Core Innovation: Style-CCL

The central contribution is Style-CCL (Multi-Stage Style Curriculum Continual Learning), designed to resolve the semantic-texture interference observed in one-stage training.

• Complexity Estimation: The authors utilize Fokker–Planck Local Intrinsic Dimensionality (FPLID) to estimate and rank the complexity of style images. They observed that semantic styles have lower LID (easier to learn) while texture styles have higher LID (harder to learn).
• Curriculum Strategy: Based on LID scores, the dataset is divided into four subsets:
  1. Semantic styles from clean data ($D_1$)
  2. Semantic styles from synthetic data ($D_2$)
  3. Texture styles from clean data ($D_3$)
  4. Texture styles from synthetic data ($D_4$)
     The model is trained sequentially from $D_1 \to D_2 \to D_3 \to D_4$, moving from "easy" (semantic) to "hard" (texture) and from "clean" to "noisy" data.
• Random Memory Rehearsal (RMR): To prevent catastrophic forgetting during sequential training, the framework employs RMR. At each stage, a fixed ratio of data is randomly sampled from previous style clusters and mixed with the current stage's data.

Key Observations

The paper identifies three critical empirical observations that motivate the proposed method:

1. Semantic-Texture Interference: Training semantic and texture styles together in one stage causes semantic styles to dominate, severely degrading texture learning.
2. Content Preservation Failure: Mixing clean and synthetic triplets in a single stage compromises the integrity of content preservation.
3. Learning Order: When trained on a single style containing both semantic and texture elements, the model learns semantic transformations early and texture transformations late, confirming that texture styles are inherently more complex.

Results

Quantitative Performance

Style-CCL achieves state-of-the-art (SOTA) performance across three core metrics compared to existing methods (e.g., OmniStyle, CSGO, StyleID):

• Style Similarity (CSD Score): 0.561 (Highest)
• Content Preservation (CPC Score): 0.401 (Highest)
• Aesthetic Quality: 6.297 (Highest)

Ablation Studies

• Multi-Stage vs. One-Stage: Moving from a one-stage mixture to a four-stage curriculum significantly improved texture style similarity (from 0.117 to 0.561) and content preservation.
• Rehearsal Rate: An optimal rehearsal rate ($R=1/3$) was found to balance style similarity and content preservation, preventing both catastrophic forgetting and the dominance of previous data.

User Study

In a study with 20 human evaluators, Style-CCL was selected as the best model in 69.75% of cases for style similarity, 30.25% for content consistency, and 72.00% for aesthetics, outperforming all baselines significantly.

Significance and Claims

The paper claims that Style-CCL successfully addresses the fundamental limitations of one-stage training for conditional DiTs. By decoupling the learning of semantic and texture styles through a curriculum approach and mitigating forgetting via random memory rehearsal, the model achieves:

1. Universal Style Capability: The ability to learn thousands of style categories without the need for specific Style LoRAs for each category.
2. High Fidelity: Preservation of complex content characteristics (facial identity, subject details) without subject confusion or mixture.
3. SOTA Performance: Establishing new benchmarks in style similarity, content consistency, and aesthetic quality.

The authors note that while the initial implementation uses FLUX-dev, the methodology has been successfully transferred to the Qwen-Image-Edit series (TeleStyle V1 and V2), achieving performance comparable to top-tier closed-source models. The paper acknowledges limitations in crowded scenes and rare, fine-grained out-of-distribution styles, noting these are addressed in subsequent TeleStyle series releases.