Technical Summary: Color-Style Disentangled Two-Stream Networks for Universal Tuning-Free Style Transfer

Problem Statement

Style transfer, while a classical computer vision problem, remains ill-defined, involving the complex transfer of attributes such as stroke, texture, material, color, light, and structure. Existing solutions face three primary limitations:

1. Inadequate Style Similarity: Current models often fail to accurately reconstruct specific style attributes like strokes or materials.
2. Limited Editing Capability: Models that excel at style reconstruction often struggle to follow text prompts or allow for content editing.
3. Color Invasion: In scenarios involving both content and style references, models frequently over-transfer the color of the style reference to the content, altering the content's inherent characteristics.

Furthermore, test-time tuning methods (e.g., StyleDrop) are computationally expensive and impractical for real-world applications requiring customized style references. While tuning-free models exist, they have not yet solved the challenge of characteristics-preserved style transfer—transferring style attributes while strictly preserving the color distribution and characteristics of the content image.

Methodology

The authors propose Color Disentangled Style Transfer (CDST), a novel two-stream training paradigm designed to completely isolate color from other style attributes. The framework operates on a single trained model capable of zero-shot inference for various style transfer tasks.

1. Training Paradigm: Two-Stream Disentanglement

CDST formulates style transfer as a conditional reference image construction task using a dual-stream architecture:

• Style Stream: To prevent color contamination, the style reference image is converted to greyscale. Features are extracted using DinoV2, utilizing a multi-feature embedding compression strategy. This involves:
  • Extracting deep grid features and the CLS token from the final block.
  • Compressing grid embeddings into 4 tokens via a Style Transformer.
  • Extracting shallow representations from Blocks 5, 11, and 17 to capture low-level cues, which are processed by MLPs and average-pooled.
  • Concatenating these 7 tokens to form the style embedding ($e_s$).
• Color Stream: Color is represented independently using a quantized color histogram. The histogram is extracted via the Rayleigh package (converting HSV to LAB space and quantizing Euclidean distances), resulting in a 180-dimensional vector. This is inflated via an MLP into 4 tokens to form the color embedding ($e_c$).

Both streams are injected into the Denoising UNet via learnable linear matrices ($W^K, W^V$) and cross-attention mechanisms. The training objective is to reconstruct the reference image using standard MSE loss.

2. Inference: UNet Disentanglement and Global Calibration

During inference, CDST employs specific strategies to balance style fidelity and editing capability:

• New Style Definition (Forgetting Mechanism): Unlike training where style tokens map to all 70 cross-attention blocks of the SDXL UNet, inference maps style tokens only to specific ranges: cross_attention_list[0:14] (Encoder) and cross_attention_list[44:70] (Decoder).
  • Weights are set to $\lambda_s = 0.2$ for the Encoder (preserving space/structure) and $\lambda_s = 0.9$ for the Decoder (preserving appearance/texture).
  • This "forgetting mechanism" prevents overfitting and content invasion, significantly enhancing editing capabilities.
• Global Color Calibration: To correct minor approximation errors from the quantized color histogram, a post-processing step converts the generated image and color reference to YUV space. It performs a statistical alignment (matching mean and variance) and blends the result back to RGB space using a hyper-parameter $\alpha$ (default 0.8).

3. CDST Workflows

The single trained model supports multiple workflows by adjusting inputs and weights:

• Characteristics-Preserved Style Transfer: Transfers style (excluding color) from a style reference while preserving the content's pixel-level characteristics. This is achieved by injecting a Content Prior Latent (noisy version of the content image) and using ControlNet (Canny edge) to maintain spatial structure. The color stream is effectively bypassed or set to zero weight, as the Content Prior Latent dominates pixel-level color.
• Style + Color + Content: Combines style, color, and content references using ControlNet.
• Style + Color + Prompt: Generates images based on a text prompt, style reference, and separate color reference.

Key Contributions

1. CDST Framework: A two-stream training paradigm that completely disentangles color from style, enabling universal tuning-free style transfer. It is the first to solve characteristics-preserved style transfer with style and content references in a tuning-free manner.
2. Improved Similarity and Editing: Through multi-feature image embedding compression (DinoV2 shallow+deep) and a new style definition based on UNet disentanglement laws, CDST achieves a superior trade-off between style similarity and editing capability compared to prior open-source and commercial models.
3. State-of-the-Art Performance: Extensive quantitative metrics and human evaluations demonstrate CDST's superiority in style similarity, prompt alignment, color consistency, and characteristics preservation across various tasks.

Results and Evaluation

The authors conducted comprehensive experiments using a benchmark of 25 style references and diverse tasks.

• Quantitative Metrics: In the "Style/Color Ref + Prompt" task, CDST achieved the best scores in prompt alignment (CLIP Score: 0.291), style similarity (CSD Score: 0.587), and color distance (0.116), outperforming InstantStyle, CSGO, and Adobe Firefly. It trailed only Midjourney v6.1 in aesthetic scores.
• Human Evaluation:
  • Style + Content: CDST won against the state-of-the-art CSGO model in style similarity (78.0%), color consistency (81.6%), and overall preference (84.4%).
  • Characteristics-Preserved Transfer: Compared to GPT-4O, CDST showed higher style similarity (63.2%) and color consistency (94.8%), though GPT-4O performed better in characteristics consistency and aesthetics. The overall user preference was nearly even (52.8% for CDST).
• Ablation Studies:
  • Embeddings: DinoV2 shallow+deep embeddings significantly outperformed CLIP embeddings and deep-only embeddings.
  • Style Definition: Mapping style tokens to specific UNet blocks (Encoder+Decoder) with forgetting mechanisms yielded better results than using all blocks or only the decoder.
  • Disentanglement: Using greyscale style references was critical; using colorful references caused confusion and color distortion when style and color references differed.

Significance and Claims

The paper claims that CDST represents a significant advancement by solving the "color invasion" problem that plagues previous tuning-free models. By isolating color via a dedicated stream and enforcing a color-blind style stream, CDST unlocks the ability to perform characteristics-preserved style transfer, a task previously unstable or impossible without fine-tuning.

The authors emphasize that CDST is a universal, tuning-free model. Despite being trained on a specific setup, it can handle diverse scenarios (Style+Prompt, Style+Content, Characteristics-Preserved) with a single model. The paper notes that while the model was trained in 2025, it remains useful in 2026, having been utilized for training data synthesis in subsequent TeleStyle models, demonstrating its robustness even as foundation models evolve toward DiT architectures.