💬 NLP

Multimodal Image Colorization: Quantifying the Impact of Text-Conditioned Guidance on Grayscale-to-Color Translation

This paper quantifies the impact of text conditioning on grayscale-to-color translation by demonstrating that integrating CLIP guidance into both U-Net and Stable Diffusion 1.5 architectures consistently improves pixel-level accuracy, perceptual similarity, and colorfulness compared to models without text input.

Original authors: Colten Reissmann, Hugo Garrido-Lestache Belinchon

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Colten Reissmann, Hugo Garrido-Lestache Belinchon

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a stack of old, black-and-white family photos. You want to bring them to life with color, but you don't remember if your grandfather's car was red or blue, or if the sky was a stormy grey or a sunny blue. This is the problem computer scientists face with image colorization: a single black-and-white image can be colored in many different, equally valid ways. It's like trying to guess the flavor of a mystery candy just by looking at its wrapper.

This paper asks a simple question: Does giving the computer a written description (a "text prompt") help it guess the right colors better?

To find out, the researchers set up a controlled experiment, like a scientific taste test, comparing two different types of "coloring chefs" (AI models):

The "Scratch" Chef (U-Net): A smaller model built from the ground up. It has to learn everything about colors from scratch, just by looking at the photos.
The "Expert" Chef (Stable Diffusion): A massive, pre-trained model that has already "seen" billions of images and knows a lot about what things usually look like.

For each chef, they created two versions:

The Silent Version: The chef looks at the black-and-white photo and tries to guess the colors on its own.
The Whispering Version: The chef looks at the photo and reads a short sentence describing the scene (e.g., "a red car" or "a blue sky").

The Results: Did the Whisper Help?

The researchers measured the results using four different "scorecards" to see how close the AI's coloring was to the real, original color photo.

1. The "Scratch" Chef (U-Net) got a huge boost.
When this smaller model was given text instructions, it improved dramatically:

Accuracy: It got about 5.6% better at matching the exact pixel colors.
Structure: It got 1.2% better at keeping the shapes and lines looking right.
Vibrancy: This was the biggest jump! The colors became 36.6% more vivid and lively.
Perception: The human eye perceived the result as 7.6% more realistic.

The Analogy: Imagine a student who has never studied art. If you just hand them a black-and-white sketch, they might color a tree brown or green randomly. But if you whisper, "This is a pine tree," they suddenly know exactly which green to use. The text gave them the missing knowledge they didn't have.

2. The "Expert" Chef (Stable Diffusion) also improved, but differently.
Since this model already knew a lot about colors, the text instructions helped, but in a more subtle way:

Accuracy: It improved by 5.8% (similar to the smaller model).
Structure: It improved by 1.5%.
Perception: It got 11.3% more realistic to the human eye.
Vibrancy: The colors only got 0.6% more vivid.

The Analogy: This chef is like a professional painter who already knows that cars are usually red or blue. If you whisper "red car," they don't need to learn what red is; they just need to be reminded to pick red instead of blue. The text didn't teach them a new skill; it just helped them make a specific choice.

Why This Matters

The paper proves that text is a powerful tool for solving the "mystery" of color.

It solves the guessing game: Without text, the AI often picks "safe," dull, or average colors (like a grey car) because it's unsure. With text, it confidently picks the specific color requested.
It works for everyone: Whether the AI is a small, learning-from-scratch model or a giant, pre-trained expert, adding text instructions consistently made the results better.
It's not just about size: The researchers showed that the improvement came specifically from the text, not from changing the model's size or structure.

The Bottom Line

Think of the black-and-white image as a puzzle with missing pieces. The text description acts like a hint that tells you exactly what those missing pieces should look like. The study confirms that giving the computer these hints results in a much clearer, more accurate, and more colorful picture, regardless of how smart the computer already is.

Note: The paper focuses strictly on measuring these improvements on a specific set of photos. It does not claim these results apply to medical imaging or other specific real-world uses, nor does it predict future technologies beyond what was tested in this specific experiment.

Technical Summary: Multimodal Image Colorization

Problem Statement

Automatic grayscale-to-color image translation is a fundamental challenge in computer vision because the task is ill-posed: a single grayscale intensity value does not uniquely determine the original color. Consequently, a single input can correspond to multiple valid colorizations depending on object identity, scene context, or artistic intent. While modern deep learning approaches utilize probabilistic models and semantic feature extraction to address this ambiguity, the inherent multimodality of the problem remains a central hurdle.

A promising strategy to resolve this ambiguity is incorporating natural language descriptions as conditioning signals. Text can explicitly define attributes (e.g., "red car" vs. "blue car") that are indistinguishable in grayscale. However, previous works integrating text conditioning (e.g., TIC, DiffColor, Diffusing Colors) simultaneously introduce multiple architectural changes, such as modified backbones, new loss functions, or diffusion frameworks. This confounds the results, making it impossible to isolate the specific quantitative contribution of text guidance from other architectural improvements.

Methodology

To isolate the impact of text conditioning, this study conducts a controlled ablation study comparing two distinct architecture tiers. In both tiers, the only variable changed is the presence of text conditioning; all other hyperparameters, datasets, and training procedures are held constant.

Dataset:
The experiments utilize the Open Image Preferences v1 dataset (7,459 prompt–image pairs). A deterministic split allocates 90% for training and 10% for evaluation, ensuring no test-set leakage.

Architecture Tier 1: U-Net

Base Model (UN-NP): A custom U-Net trained from scratch on grayscale inputs to predict RGB outputs. It uses MSE loss and contains ~13.4M parameters.
Text-Conditioned (UN-P): An identical backbone augmented with cross-attention layers at specific encoder/decoder depths. These layers attend to embeddings from a frozen CLIP ViT-B/32 encoder. The model contains ~16.4M parameters.
Training: Both models use Adam optimization. UN-P is trained with 10% empty prompt dropout to prevent overfitting to text.

Architecture Tier 2: Stable Diffusion 1.5

Base Model (SD-NP): A fine-tuned Stable Diffusion 1.5 checkpoint where every caption is replaced with an empty string, effectively training an unconditional model.
Text-Conditioned (SD-P): The same checkpoint fine-tuned with real captions.
Configuration: Both variants operate in CIELAB space (predicting ab channels). Crucially, both use a guidance scale of 1.0 (no Classifier-Free Guidance amplification). This design choice isolates the effect of text conditioning through cross-attention alone, avoiding the risk of text signals overwhelming grayscale conditioning, which was observed in earlier high-scale experiments.

Evaluation Metrics:
Performance is measured on a held-out test set of 745 images using:

PSNR & SSIM: Pixel-level and structural fidelity.
LPIPS: Perceptual similarity (lower is better).
Colorfulness: Hasler–Süsstrunk index (higher is more vivid).
All metrics are computed at 512×512 resolution (U-Net predictions are bilinearly upsampled for fair comparison).

Key Contributions

Controlled Ablation: The first study to quantify the impact of text conditioning on colorization by holding all other variables constant across two vastly different architecture scales (a ~13M parameter U-Net and an ~859M parameter Stable Diffusion model).
Quantitative Evaluation: A rigorous statistical analysis using paired t-tests on per-image metrics to determine the significance of improvements.
Qualitative Analysis: An examination of specific scenarios where text conditioning provides the most benefit (disambiguating ambiguous objects) versus where it offers limited advantage (stereotypical scenes).

Results

The study demonstrates that text conditioning provides consistent, measurable improvements across both architecture scales.

U-Net Tier (UN-NP vs. UN-P):

PSNR: +5.6% improvement (19.13 → 20.21).
SSIM: +1.2% improvement.
LPIPS: -7.6% reduction (improvement).
Colorfulness: +36.6% increase.
Significance: All improvements are highly statistically significant ( $p < 10^{-13}$ ) with medium effect sizes.

Stable Diffusion Tier (SD-NP vs. SD-P):

PSNR: +5.8% improvement (19.65 → 20.79).
SSIM: +1.5% improvement.
LPIPS: -11.3% reduction (improvement).
Colorfulness: +0.6% increase (statistically insignificant, $p=0.71$ ).
Significance: PSNR, SSIM, and LPIPS improvements are highly significant.

Comparative Analysis:
While PSNR gains are comparable across tiers (~5.6–5.8%), the nature of the improvement differs. The U-Net tier shows a dramatic increase in colorfulness, suggesting text provides a critical external source of semantic information for models trained from scratch. In contrast, the Stable Diffusion tier, which possesses rich pretrained visual priors from billions of text-image pairs, shows a larger reduction in perceptual error (LPIPS) but minimal gain in colorfulness, as the backbone already encodes strong color associations.

Significance and Claims

The paper concludes that text conditioning is an architecture-independent mechanism that improves colorization quality. The consistency of PSNR and LPIPS improvements across a 50× difference in parameter count suggests that the benefit is not merely a function of model capacity but a fundamental resolution of the ill-posed nature of the problem.

The authors argue that for models trained from scratch (like the U-Net), text conditioning acts as a necessary signal to resolve color ambiguity that grayscale pixels alone cannot provide, effectively breaking an information bottleneck rather than a capacity bottleneck. For large pretrained models (Stable Diffusion), text conditioning refines the output to be more semantically accurate and perceptually closer to the ground truth, even if the marginal gain in color vividness is lower.

The study emphasizes that previous claims of text-guided superiority were difficult to attribute solely to text due to concurrent architectural changes. By isolating the variable, this work confirms that natural language guidance is a distinct and powerful contributor to automatic image colorization.