💻 computer science

NumColor: Precise Numeric Color Control in Text-to-Image Generation

NumColor is a novel framework that enables precise numerical color control in text-to-image generation by introducing a learnable ColorBook and specialized training losses to overcome tokenization limitations, achieving significant accuracy and harmony improvements across multiple diffusion models without requiring model-specific adaptation.

Original authors: Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Atif Butt, Diego Hernandez, Alexandra Gomez-Villa, Kai Wang, Javier Vazquez-Corral, Joost Van De Weijer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very talented artist who can paint anything you describe. If you say, "Paint a red car," they instantly understand. But if you try to be precise and say, "Paint a car in the exact shade of red with the code #FF5733," the artist gets confused. They might squint at the code, break it into random pieces like "#", "FF", "57", and "33", and then paint a rainbow car, a blue car, or just ignore the color entirely.

This is the problem the paper NumColor solves. It's like giving the artist a new, super-precise dictionary and a translator so they can finally understand exact color codes.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Broken Puzzle"

Current AI image generators (like FLUX or Stable Diffusion) read text using a system called tokenization. Think of this like a puzzle where words are broken into small chunks.

The Issue: When you type a color code like #FF5733, the AI doesn't see it as "one color." It sees it as four separate, meaningless puzzle pieces: #, FF, 57, and 33.
The Result: The AI's brain (the text encoder) doesn't know what these pieces mean together. It's like trying to understand a sentence where every word is a different language. The AI gets lost, and the color comes out wrong.

2. The Solution: The "Color Translator" (NumColor)

The authors built a tool called NumColor that acts as a bridge between the confusing code and the artist's brain. It has two main parts:

Part A: The "Color Token Aggregator" (The Detective)

Imagine a detective who walks into a room full of scattered puzzle pieces.

What it does: This module scans the text and says, "Hey! These four pieces (#, FF, 57, 33) actually belong together. They form one single concept: a specific color."
The Magic: It glues those scattered pieces back into one coherent "color token" before the AI tries to understand it. It doesn't matter if the AI usually breaks words differently; this detective finds the color no matter how it's chopped up.

Part B: The "ColorBook" (The Master Palette)

Now that the AI has one clean "color token," it still needs to know what color that token represents.

The Problem: The AI's brain doesn't have a specific "slot" for #FF5733. It only knows general words like "red" or "blue."
The Solution: The authors created a ColorBook. Think of this as a massive, magical library containing 6,707 specific color cards.
- Instead of trying to teach the AI 16 million possible colors (which is impossible), they picked the most important 6,707 "anchor" colors that cover the whole spectrum.
- When the AI sees a new code, it looks up the closest "anchor" in the ColorBook.
- The Smoothness: If you ask for a color that isn't exactly in the book (like a color right between two cards), the system uses soft interpolation. Imagine blending two paints on a palette; the AI smoothly mixes the two closest "anchor" colors to create the exact shade you asked for.

3. The Training: The "Perfect Studio"

To teach this system, they couldn't just use photos from the internet. Why? Because in real photos, lighting, shadows, and materials make a "red" apple look different than a "red" car. The AI would get confused.

The Fix: They built a synthetic studio (called NumColor-Data).
They took 3D models of objects (cars, vases, teddy bears) and painted them with exact mathematical colors in a computer.
They generated 500,000 images where the color on the screen matches the code in the text perfectly, with no shadows or weird lighting to confuse the AI. This gave the AI a "textbook" example of what every color code actually looks like.

4. The Results: From "Rainbow" to "Precision"

Before this tool, if you asked an AI for a specific hex code, it was right less than 15% of the time.

After NumColor: The accuracy jumped by 4 to 9 times.
The Best Part: They trained this tool on one specific AI model (FLUX), but because the "translator" and the "ColorBook" are separate add-ons, they can plug them into other AI models (like SD3 or PixArt) without retraining them. It's like giving a universal remote control to different brands of TVs.

Summary Analogy

Think of the AI as a chef who is great at cooking "spicy food" or "sweet food" but terrible at reading a recipe that says "add 3.42 grams of salt."

The Old Way: The chef reads "3.42" as "three," "point," "four," "two," and adds a random amount of sugar instead.
The NumColor Way:
1. The Detective (Aggregator) realizes "3.42" is a single measurement unit.
2. The Master Scale (ColorBook) translates "3.42 grams" into the exact pinch of salt the chef needs.
3. The Perfect Kitchen (Synthetic Data) taught the chef exactly what that amount of salt looks like.

Now, the chef can cook with exact precision, turning any specific color code into a perfect image, just like a professional designer would expect.

1. Problem Statement

Current Text-to-Image (T2I) diffusion models excel at generating images from natural language but fail significantly when conditioned on precise numerical color specifications (e.g., Hex codes like #FF5733 or RGB values like rgb(255,87,51)).

The paper identifies two root causes for this failure:

Subword Tokenization Fragmentation: Standard text encoders (like CLIP and T5) fragment numerical color codes into arbitrary, semantically meaningless sub-tokens (e.g., #, FF, 57, 33). These fragments lack coherent color semantics in the embedding space and disrupt cross-attention mechanisms, causing the model to attend to unrelated image regions rather than the target object.
Alignment Gap: Even if tokens were unified, there is no pre-trained mapping between numerical color representations and the text encoder's embedding space. Existing training data (e.g., LAION) relies on color names (e.g., "red") rather than precise codes, leaving models unable to learn the geometric relationship between specific numerical values and visual output.

Consequently, state-of-the-art models achieve less than 15% accuracy on numerical color tasks, rendering them unsuitable for professional applications requiring strict brand fidelity (e.g., advertising, product visualization).

2. Methodology: NumColor Framework

The authors propose NumColor, a plug-and-play framework comprising two main components and a synthetic dataset, designed to work across diverse diffusion architectures without retraining the base models.

A. Color Token Aggregator (CTA)

To solve the tokenization fragmentation issue, the authors introduce a tokenizer-agnostic module.

Architecture: A lightweight character-level sequence labeler inspired by Named Entity Recognition (NER). It processes tokens as character sequences (up to 32 chars) using parallel 1D convolutions (kernel sizes 2, 3, 4) and a Transformer encoder.
Function: It detects numerical color spans (Hex or RGB) regardless of how the underlying tokenizer splits them. It uses a CRF (Conditional Random Field) layer to output valid B (Begin), I (Inside), O (Outside) tags, unifying fragmented tokens into a single semantic span.
Output: The detected span is converted into CIE Lab coordinates for lookup.

B. ColorBook

To bridge the alignment gap, the authors construct a learnable codebook that maps colors to the text encoder's embedding space.

Design: Instead of learning embeddings for all 16.7 million RGB colors, they sample 6,707 anchor colors uniformly from the perceptually uniform CIE Lab space within the sRGB gamut (5-unit intervals).
Embedding Injection: Each anchor maps to a learnable embedding vector ( $d=4096$ ). Crucially, these embeddings are injected pre-contextualization (before the T5 self-attention layers). This allows the color embedding to participate in bidirectional attention with object tokens (e.g., #FF5733 $\leftrightarrow$ car), ensuring proper color-object binding.
Soft Interpolation: For colors not present as anchors, the system uses soft nearest-neighbor interpolation over the $k=8$ nearest anchors in Lab space, weighted by a temperature parameter ( $\tau$ ). This ensures smooth transitions across the color gamut.
Initialization: Embeddings are initialized using ISCC-NBS color category centroids and CSS3/X11 names to avoid semantic contamination (e.g., avoiding "salmon" which implies fish, not just a color).

C. Training Strategy & Losses

The framework is trained in two stages:

CTA Training: Trained on 200k synthetic prompts (color names replaced by codes) to learn token detection.
ColorBook Training: The base diffusion model (FLUX) and text encoders are frozen. Only the ColorBook embeddings are updated using NumColor-Data (see below).
- Primary Loss: Flow Matching Loss (reconstruction of the image).
- Auxiliary Losses:
  - Directional Alignment: Ensures the vector difference between two color embeddings in the text space aligns with their difference in Lab space.
  - Interpolation Consistency: Enforces that the embedding of a midpoint color in Lab space equals the midpoint of the embeddings of the two endpoint colors.

3. Key Contributions

NumColor Framework: A novel architecture enabling precise numerical color control in T2I models by solving tokenization fragmentation and embedding misalignment. It is the first method to handle arbitrary Hex/RGB codes without model-specific retraining.
NumColor-Data: A large-scale synthetic dataset of 500,000 images. It renders 3D objects with Lambertian materials and controlled lighting, providing unambiguous pixel-to-color correspondence, eliminating the noise inherent in photographic datasets.
Zero-Shot Transferability: The ColorBook, trained solely on the FLUX architecture, transfers zero-shot to SD3, SD3.5, PixArt-α, and PixArt-Σ without any fine-tuning of the base models, demonstrating universality across different text encoders (CLIP/T5 combinations).

4. Results

Evaluated on the GenColorBench benchmark (9.8k prompts, 400+ colors):

Color Accuracy: NumColor improves numerical color accuracy by 4–9× across five models.
- Example: On FLUX, accuracy on ISCC-L1 (coarse) jumps from 13.82% to 55.71%. On fine-grained CSS3/X11 tasks, it improves from ~5.8% to 51.96%.
Color Harmony: Improves aesthetic coherence by 10–30×, reducing color harmony scores (lower is better) from hundreds to the 16–26 range.
Qualitative Performance:
- Successfully generates fine-grained gradients (e.g., grayscale, blue-to-cyan) with $\Delta E < 15$ .
- Correctly handles interpolated colors not present in the training anchors.
- Fixes "rainbow artifacts" and scattered attention maps seen in baseline models.
Embedding Geometry: Analysis shows that NumColor reorganizes the embedding space to preserve perceptual structure (Lab geometry) while maintaining compatibility with the frozen T5 encoder.

5. Significance and Limitations

Significance:

Professional Applicability: This work removes a critical barrier for using generative AI in design, branding, and e-commerce, where exact color matching (Hex/RGB) is mandatory.
Efficiency: It achieves these results without retraining massive diffusion backbones, relying instead on a small, efficient codebook and aggregator.
Generalization: It proves that color control can be decoupled from specific model architectures, working across the spectrum of modern Diffusion Transformers.

Limitations:

Color Leakage: In complex scenes, the specified color may bleed into surrounding elements due to a lack of explicit spatial grounding in the attention mechanism.
Memory Color Bias: Objects with strong canonical associations (e.g., sunflowers, bananas) may resist recoloring, as the model's pre-trained priors override the injected color embedding.

In conclusion, NumColor represents a significant leap forward in controllable generation, transforming T2I models from tools that understand "red" to tools that can precisely render #FF5733.