💻 computer science

OmniColor: A Unified Framework for Multi-modal Lineart Colorization

OmniColor is a unified framework for multi-modal lineart colorization that integrates spatially-aligned and semantic-reference guidance through specialized encoding strategies and an adaptive gating module to achieve precise, flexible, and temporally stable results under diverse user constraints.

Original authors: Xulu Zhang, Haoqian Du, Xiaoyong Wei, Qing Li

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xulu Zhang, Haoqian Du, Xiaoyong Wei, Qing Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a professional animator or comic book artist. You have a beautiful, black-and-white sketch (called lineart) of a character. It's perfect in shape, but it's empty. Your job is to fill it with color.

In the old days, you had to paint every single pixel by hand. Then, computers came along and tried to help, but they were like clumsy assistants: they could guess the colors, but they often messed up the details, forgot what the character looked like in the next frame, or didn't listen when you said, "Make the hair blue, not green."

OmniColor is like hiring a super-intelligent, multi-talented art director who can listen to any instruction you give and combine them perfectly. Here is how it works, broken down into simple concepts:

1. The Two Types of Instructions

The paper realizes that artists give instructions in two very different ways. OmniColor treats them like two different types of mail:

The "Blueprint" (Spatially-Aligned): This is like a strict architectural drawing. It says, "The line here must stay exactly here," or "This specific spot must be red."
- Examples: The original sketch, a quick scribble of color you made, or the frame from the movie just before this one.
- How OmniColor handles it: It uses a Dual-Path Strategy. Imagine two eyes: one eye looks at the fine details (the edges of the lines) to make sure nothing blurs, and the other eye looks at the big picture to understand the scene. They work together to make sure the colors stay exactly where you want them.
The "Vibe Check" (Semantic-Reference): This is like giving a mood board or a description. It says, "Make it look like a spooky night" or "This character should look like a superhero." It doesn't care about exact pixel locations; it just wants the feeling to be right.
- Examples: Text prompts ("sunset"), a photo of a character's face to copy their look, or old frames from way back in the movie.
- How OmniColor handles it: It uses a Smart Filter. Instead of trying to memorize every single pixel of a reference photo (which is slow and wasteful), it uses a "Vision-Language Model" (a super-smart AI that understands pictures and words) to grab just the essence of the reference. It throws away the boring, repetitive parts (like a background that doesn't change) to save time.

2. The "Traffic Cop" (Adaptive Gating)

Sometimes, your instructions might fight each other.

Scenario: You tell the AI, "Make the sky blue" (a text instruction), but you also scribbled "red" on the sky in your sketch (a spatial instruction).
The Problem: A normal AI gets confused and makes a purple mess.
OmniColor's Solution: It has a Traffic Cop (called the AS-Gate). This little module looks at your instructions and decides which one is more important right now. If your sketch is very detailed, the Traffic Cop says, "Listen to the sketch!" If your sketch is vague, it says, "Okay, listen to the text description." This prevents the AI from getting confused and making weird, bleeding colors.

3. The "Time Machine" (Sequential Generation)

In animation, characters move. If you color Frame 1, then Frame 2, then Frame 3, the character's shirt might change color from blue to green to purple just because the AI forgot what it did a second ago.

OmniColor's Solution: It remembers the past. It looks at the previous frame (the "recent history") to keep the colors consistent. It also has a Redundancy Eliminator that says, "Hey, the background hasn't changed at all in these last 10 frames; I don't need to re-calculate that." This makes the process super fast and keeps the movie looking smooth.

Why is this a big deal?

Think of other AI art tools as single-purpose tools. One tool is great at following text prompts, another is great at copying a reference photo, and another is good at following a sketch. But if you want to do all three at once? They usually crash or produce garbage.

OmniColor is like a Swiss Army Knife for animators. It can handle:

A sketch + a text prompt + a reference photo + a previous frame all at the same time.
It knows when to be strict (sticking to the lines) and when to be creative (following the mood).
It's fast enough to be used in real professional studios.

In short: OmniColor is the first AI that truly understands that coloring a movie isn't just about picking colors; it's about listening to a mix of strict rules, creative vibes, and memory, all while keeping the character looking the same from start to finish. It turns a chaotic pile of instructions into a beautiful, consistent masterpiece.

1. Problem Statement

Lineart colorization is a critical step in professional animation, comic, and game design. While existing methods can generate colored images from line drawings, they face significant challenges in real-world production environments:

Limited Flexibility: Most frameworks support only a single type of control signal (e.g., text or reference image), making them unsuitable for complex scenes requiring multiple, overlapping constraints.
Inconsistency: Maintaining character identity and stylistic continuity across a sequence of frames is difficult, especially when relying solely on structural lineart.
Conflict Resolution: When multiple inputs (e.g., a global style reference vs. a local color hint) contradict each other, existing models often fail to prioritize the correct guidance, leading to artifacts like color bleeding or loss of structural fidelity.
Efficiency: Processing long animation sequences often involves redundant information, leading to high computational costs and slow inference.

2. Methodology

OmniColor proposes a unified framework based on a Multi-modal Diffusion Transformer (MMDiT) backbone, optimized via Flow Matching. The core innovation lies in systematically categorizing input signals and processing them through specialized modules.

A. Signal Categorization

The framework divides guidance signals into two distinct types:

Spatially-aligned Conditions ( $C_{spat}$ ): Inputs requiring strict pixel-level alignment with the output geometry.
- Examples: Input lineart, user-provided sparse color hints, and recent history frames.
Semantic-reference Conditions ( $C_{sem}$ ): Inputs providing high-level guidance without requiring exact spatial correspondence.
- Examples: Text prompts, character identity (ID) reference maps, and long-term history frames.

B. Key Architectural Components

Dual-Path Encoding Strategy (for $C_{spat}$ ):
- Uses a VAE encoder to preserve high-frequency details and precise color boundaries.
- Uses a Vision-Language Model (VLM) encoder to extract dense, localized semantic tokens for scene layout understanding.
- Dense Feature Alignment (DFA) Loss: To ensure the generated latent space matches the ground truth in high-frequency detail, the model applies a loss function using DINOv3 features during the late stages of the denoising process ( $t \to 1$ ). This enforces sharp boundaries and reduces artifacts.
VLM-Only Encoding & Temporal Redundancy Elimination (for $C_{sem}$ ):
- VLM-Only: Abstract references (text, ID maps) are processed solely by the VLM encoder to avoid the computational overhead of generating dense feature maps for every frame.
- Temporal Redundancy Elimination (TRE): For animation sequences, this module analyzes consecutive frames, identifies redundant patches based on color histogram similarity, and generates a compact "delta-content" image. This significantly shortens the token sequence, accelerating inference while maintaining long-term consistency.
Adaptive Spatial-Semantic Gating (AS-Gate):
- To resolve conflicts between inputs (e.g., a text prompt contradicting a color hint), the AS-Gate module dynamically modulates the influence of the semantic branch based on the information density of the spatial branch.
- It uses a LoRA-based structure with zero-initialization, allowing the model to learn a scaling factor that balances spatial constraints and semantic creativity during training.

3. Key Contributions

Unified Multi-modal Framework: The first framework to support arbitrary combinations of heterogeneous control signals (Lineart, Text, Color Hints, ID References, and Temporal History) in a single model.
Novel Signal Processing: Introduction of a dual-path encoding strategy for spatial inputs and a VLM-only + TRE strategy for semantic inputs, optimizing both fidelity and efficiency.
Conflict Resolution Mechanism: The AS-Gate module effectively handles conflicting instructions, ensuring the model prioritizes explicit user guidance (like color hints) over global semantic descriptions when necessary.
High-Fidelity Structural Preservation: The integration of DFA loss with DINOv3 supervision ensures that the generated colors strictly adhere to the lineart boundaries without blurring.

4. Experimental Results

The authors evaluated OmniColor on a dataset of 120,000 pairs from 25 animation series, testing against state-of-the-art (SoTA) methods like ControlNet, MagicColor, and unified generation models (e.g., Flux, Seedream).

Quantitative Performance:
- OmniColor outperforms all competitors across metrics including Image-Align (0.9246), SSIM (0.7611), PSNR (16.38), and FID (91.49).
- It demonstrates superior Temporal Consistency, reducing the Consistency Error ( $\Delta FC$ ) to 0.37 in sequential generation tasks, compared to much higher errors in other methods.
Efficiency:
- The TRE mechanism reduces token overhead by ~30% (from 7,521 to 5,203 tokens).
- Inference time is reduced from 452s to 152s per sample compared to a baseline without TRE and AS-Gate optimizations.
User Study:
- In a study with 10 professional evaluators, OmniColor achieved a 61.0% overall preference rate, significantly outperforming the second-best method (11.6%).
- It dominated in Structural Fidelity (81.0%), proving its ability to respect lineart boundaries better than any other model.

5. Significance

OmniColor represents a significant leap forward for AI-assisted creative workflows in the animation and illustration industries. By bridging the gap between rigid structural constraints and flexible semantic intent, it offers a scalable, practical solution for:

Professional Production: Enabling artists to use mixed-modal controls (e.g., "Color this character like this reference, but change the shirt to blue") without retraining or switching models.
Temporal Stability: Solving the flickering and identity drift issues common in video generation, making it suitable for full-length animation sequences.
Efficiency: Providing a fast inference pipeline suitable for iterative editing and real-time creative exploration.

The paper concludes that by decoupling spatial and semantic processing and introducing adaptive gating, OmniColor sets a new standard for controllable, high-fidelity lineart colorization.