💻 computer science

TimeColor: Flexible Reference Colorization via Temporal Concatenation

TimeColor is a flexible sketch-based video colorization model that enhances color fidelity, identity consistency, and temporal stability by encoding heterogeneous, variable-count references as temporally concatenated latent frames with explicit region assignment and spatiotemporal correspondence-masked attention.

Original authors: Bryan Constantine Sadihin, Yihao Meng, Michael Hua Wang, Matteo Jiahao Chen, Hang Su

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bryan Constantine Sadihin, Yihao Meng, Michael Hua Wang, Matteo Jiahao Chen, Hang Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an animation director. You have a stack of black-and-white pencil sketches for a new cartoon episode. Your job is to turn these sketches into a vibrant, colorful movie.

In the past, the "colorist" (the person or AI doing the coloring) had a very strict rule: They could only look at one single picture to decide what colors to use. Usually, this was the very first frame of the scene.

The Problem:
This is like trying to paint a whole house by looking at a photo of just the front door.

If the character turns around, the AI gets confused.
If the character walks into a new scene, the AI might forget what color their shirt was.
If you have a "character sheet" (a drawing showing the character from the front, side, and back) or a background painting, the old AI couldn't use them. It was stuck with just that one first frame.
Sometimes, the AI would get lazy and "cheat" by copying colors from the wrong character (e.g., making the villain's hair the same color as the hero's).

The Solution: TimeColor
The researchers built a new AI called TimeColor. Think of it as a super-smart colorist who can look at many different reference pictures at the same time and knows exactly which part of the sketch belongs to which picture.

Here is how it works, using some fun analogies:

1. The "Train Car" Analogy (Temporal Concatenation)

Imagine the AI's brain is a long train.

Old Way: The train could only carry the "sketch" cars and one "reference" car. If you wanted to add more reference photos (like a character sheet or a background), you had to build a whole new, bigger train (which is expensive and slow).
TimeColor Way: TimeColor builds a train that can stretch as long as needed. It takes your sketch video and stacks all your reference photos (the first frame, a random frame from later, a character sheet) onto the same train tracks, one after another.
The Magic: The AI engine (the locomotive) stays the same size and cost. It just pulls a longer train. It looks at the sketch and all the reference photos simultaneously, step-by-step, without needing to be retrained for every new photo you add.

2. The "Name Tag" System (Modality-Disjoint RoPE)

When you put a sketch, a reference photo, and a random frame all on the same train, they might get mixed up. The AI might think the background of the reference photo is part of the character's face.

The Fix: TimeColor gives every type of picture a different "Name Tag" (a special code).
- Sketches get a "Sketch" tag.
- References get a "Reference" tag.
- The target video gets a "Target" tag.
This ensures the AI knows, "Oh, this pixel is from the reference photo, and that pixel is from the sketch. They are different things, even though they are in the same room."

3. The "Strict Bouncer" (Correspondence-Masked Attention)

This is the most important part. Imagine you have a character named "Bob" and a reference photo of "Bob's red hat." You also have a reference photo of "Alice's blue hat."

The Problem: Without a strict rule, the AI might look at Bob's sketch, see a hat, and accidentally grab the blue hat from Alice's photo because it looks "similar" in shape. This is called "leakage."
The Fix: TimeColor uses a Strict Bouncer. Before the AI is allowed to look at a reference photo to pick a color, the Bouncer checks a map.
- The map says: "The pixel at (x, y) on the sketch belongs to Bob. Therefore, you are only allowed to look at Bob's reference photo for color."
- The AI is physically blocked from looking at Alice's photo when coloring Bob. This prevents the colors from "bleeding" into the wrong characters.

Why is this a big deal?

In the real world of animation, artists rarely work with just one picture. They have:

Character Sheets: To see the character from all angles.
Background Paintings: To get the lighting and mood right.
Random Frames: Maybe they want the character to look like they did in a scene from 10 minutes ago, not the first second of the current scene.

TimeColor is like giving the AI a full toolbox instead of just a single screwdriver. It can handle a variable number of reference photos, mix and match them, and keep the colors consistent, all without needing a bigger, more expensive computer.

The Result:
The paper shows that TimeColor creates cartoons that look more professional, keep the characters looking like themselves (even when they move or turn), and don't accidentally swap colors between different people. It's a huge step toward making AI animation tools that actually feel like they understand the job, rather than just guessing.

1. Problem Statement

Current video colorization models, particularly those based on diffusion, face significant limitations in production workflows:

Single-Reference Dependency: Most existing methods condition generation on a single reference image (typically the first frame of the scene). This fails to utilize other valuable sources like character design sheets, background paintings, or arbitrary colorized frames from different shots.
Cross-Identity Leakage & Shortcutting: When multiple references are introduced, models often suffer from "shortcutting," where the model ignores explicit region assignments and instead matches sketch regions to the most visually similar reference. This leads to cross-identity palette leakage (e.g., a character's hair taking the color of a different character's shirt).
Inflexibility: Existing solutions often rely on channel stacking or adapter branches that assume a fixed number of inputs, making them unsuitable for variable-count, heterogeneous reference sets (e.g., mixing a character sheet with an arbitrary frame).
Data Scarcity: There is a lack of large-scale datasets with reliable instance tracking and pixel-level correspondence masks required to train multi-reference colorization models.

2. Methodology: TimeColor

TimeColor is a Diffusion Transformer (DiT)-based framework designed to handle variable-count, heterogeneous multi-reference conditioning while maintaining a fixed parameter budget.

A. Core Architecture: Temporal Concatenation

Instead of using channel stacking or adapter modules, TimeColor encodes all references (sketches, character sheets, arbitrary frames) as additional latent frames.

Mechanism: The target video, sketch video, and all reference images are embedded into token grids and concatenated along the temporal axis.
Benefit: This allows an arbitrary number of references to be processed concurrently in every diffusion step without increasing the model's parameter count. The sequence length changes, but the model weights remain fixed.

B. Key Technical Innovations

To prevent the model from ignoring reference assignments or mixing identities, TimeColor introduces two critical mechanisms:

Modality-Disjoint RoPE (Rotary Position Embedding):
- Naive concatenation causes positional interference between different modalities (target, sketch, reference).
- TimeColor assigns disjoint index ranges to different modalities. Reference tokens are given distinct negative temporal indices ( $l = -r$ ), while target and sketch tokens share positive indices. This prevents positional confusion and ensures the model distinguishes between the "content" being generated and the "reference" guiding it.
Spatiotemporal Correspondence-Masked Attention:
- The Problem: Standard self-attention allows tokens to attend to any other token, leading to cross-reference leakage.
- The Solution: A hard attention mask is applied. Each spatiotemporal patch in the target/sketch is assigned a specific reference identity ( $\rho$ ) via a correspondence mask.
- Constraint: A query token is only allowed to attend to key/value tokens if they share the same reference identity (or if the token is non-reference). This enforces explicit subject-reference binding, preventing the model from "cheating" by looking at the wrong reference.

C. Automated Dataset Pipeline

To train this model, the authors created a large-scale dataset from the Sakuga-42M collection using an automated pipeline:

Tools: InternVL3 (for subject enumeration), Grounding DINO (for detection), and SAM2 (for tracking).
Process: The system iteratively detects main subjects, propagates masks across frames, and filters references to ensure they come from different scenes or time gaps (to avoid trivial copying).
Output: Generates per-instance RGB references, background references, and dense pixel-level correspondence masks.

3. Key Contributions

TimeColor Framework: A DiT-based video colorization model supporting heterogeneous, variable-count references via temporal concatenation, eliminating the need for parameter-heavy adapters.
Novel Attention Mechanisms: The combination of Modality-Disjoint RoPE and Spatiotemporal Correspondence-Masked Attention effectively solves the reference shortcutting and identity leakage problems common in multi-reference generation.
Large-Scale Data Pipeline: An automated system for generating multi-reference sketch video colorization data with instance tracking and correspondence masks, addressing a critical data bottleneck in the field.

4. Experimental Results

The model was evaluated on the Sakuga-42M test set under three regimes: Starting-Frame, Arbitrary-Frame, and Multi-Reference.

Quantitative Performance: TimeColor outperformed state-of-the-art baselines (LVCD, AniDoc, ToonCrafter, ToonComposer, LongAnimation, VACE) across all metrics (SSIM, PSNR, LPIPS, FVD, FID).
- Notable Gain: The largest improvements were seen in the Multi-Reference setting, where TimeColor achieved an SSIM of 0.7589 compared to the next best (AniDoc) at 0.5798.
Qualitative Performance:
- Fidelity: Better preservation of color palettes and edge adherence.
- Consistency: Strong temporal stability and identity consistency across shots, even with large pose/viewpoint gaps between references and targets.
- Robustness: The model remained effective even when using imperfect masks (propagated via SAM2 or morphed), demonstrating resilience to annotation noise.
Ablation Studies:
- Removing Modality-Disjoint RoPE led to washed-out colors and instability.
- Removing Correspondence Masking (using full attention) resulted in significant cross-reference palette leakage.
- Soft conditioning (concatenating masks as an extra input stream) failed to prevent shortcutting, proving that hard gating is necessary.

5. Significance

TimeColor represents a significant step forward in production-ready generative AI for animation.

Workflow Efficiency: It allows animators to use diverse, reusable assets (character sheets, background art) rather than being forced to colorize frame-by-frame or rely on a single keyframe.
Scalability: By keeping the parameter count fixed regardless of the number of references, the model is computationally efficient and scalable to complex scenes with many characters.
Control: The explicit region-level binding provides a level of control previously unattainable in diffusion-based video colorization, solving the "identity leakage" problem that has plagued multi-reference generation.

In summary, TimeColor bridges the gap between academic diffusion models and industrial animation needs by offering a flexible, robust, and high-fidelity solution for sketch-to-video colorization using multiple, heterogeneous references.