Uni-Animator: Towards Unified Visual Colorization

Uni-Animator is a novel Diffusion Transformer-based framework that unifies image and video sketch colorization by introducing instance patch embedding for precise color transfer, physical feature reinforcement for high-frequency detail preservation, and sketch-based dynamic RoPE encoding to ensure temporal coherence in large-motion scenes.

The Gist

Imagine you have a black-and-white sketchbook. Maybe it's a drawing of a dragon, or a sequence of frames showing a character running. Now, imagine you want to bring these drawings to life with color, just like a professional animator would.

Doing this by hand is a nightmare. You'd have to sit there for hours, carefully filling in every line, making sure the dragon's scales look shiny and the character's shirt stays the same shade of red in every single frame of the video. If you make a mistake in frame 10, you might have to fix frames 11 through 100 to keep it consistent.

Uni-Animator is like a super-smart, magical assistant that does all this work for you in seconds. But it's not just a simple "fill bucket" tool; it's a unified brain that understands both single drawings and moving videos, and it solves three specific problems that other AI tools struggle with.

Here is how it works, explained with some everyday analogies:

1. The Problem: The "Blurry Photocopier" Effect

Most existing AI tools for coloring sketches are like a photocopier that smudges the ink.

• The Issue: When they try to color a video, the colors might "flicker" (like a bad TV signal) or the character might look like they are slipping on ice because the AI doesn't understand how the drawing is moving. Also, they often lose the tiny details, like the texture of a leather jacket or the sparkle in an eye, turning them into a smooth, plastic-looking blob.
• The Uni-Animator Fix: It acts like a high-definition restoration artist. Instead of just guessing the color, it looks at the "physical" properties of the drawing (like how light hits a surface) to keep those sharp edges and textures intact.

2. The Problem: The "Confused Tourist" Effect

If you show an AI a picture of a red car and ask it to color a sketch of a car, it might turn the sketch blue because it got confused by the lighting or the angle.

• The Issue: Old methods look at the reference image as a whole "blob" of color. They miss the specific details, like "the wheels are black" or "the roof is white."
• The Uni-Animator Fix: This tool uses something called Instance Patch Embedding. Think of this as giving the AI a magnifying glass and a sticky note. Instead of looking at the whole reference photo, it breaks the photo into tiny pieces (patches), labels them ("this is the red coat," "this is the blue hair"), and sticks those labels directly onto the sketch. This ensures that if you reference a red coat, the sketch gets a red coat, not a red hat.

3. The Problem: The "Stuttering Dancer" Effect

When a character in a video moves quickly, the AI often gets lost. It might color the character's arm in one spot in frame 1, and then suddenly jump to a different spot in frame 2, causing the video to jitter or "flicker."

• The Issue: Standard AI tools use a fixed rulebook for how things move in time. But real life (and sketches) are messy. Sometimes a character runs fast horizontally but barely moves up and down. A fixed rulebook can't handle that.
• The Uni-Animator Fix: It uses Sketch-Based Dynamic RoPE. Imagine a conductor leading an orchestra.
  • If the music is slow and steady (a static scene), the conductor keeps a slow, steady beat.
  • If the music speeds up (a character running fast), the conductor instantly changes the tempo to match the speed.
  • Uni-Animator does this by looking at the sketch's motion. If the character is moving fast to the right, it speeds up the "time clock" for that specific part of the image to keep the color locked onto the character. If they are standing still, it slows the clock down to keep the image stable. This stops the flickering.

Why is "Unified" a Big Deal?

Before this, you needed two different tools: one for coloring a single picture and a completely different, complex tool for coloring a video.

• Uni-Animator is like a Swiss Army Knife. It's one single tool that handles both. Whether you have one sketch or a 100-frame animation, it uses the same brain. This saves time and money for animators and game developers who need to switch between static art and moving scenes constantly.

The Result

In simple terms, Uni-Animator takes your black-and-white sketches and:

1. Reads your reference photos with a magnifying glass to get the colors exactly right.
2. Preserves the "crunchy" details (like fabric texture) so it doesn't look like plastic.
3. Dances with the motion, adjusting its speed to match how fast your character is moving, so the video looks smooth and professional, not jittery.

It turns the tedious, hours-long job of an animator into a quick, automated process that still feels like it was done by a human artist.

Technical Summary

1. Problem Definition

The paper addresses the challenge of sketch colorization for both static images and dynamic videos. While existing methods have made progress, they suffer from three critical limitations that hinder unified, high-fidelity production:

• Domain Specialization: Current models are typically specialized for either static images or dynamic videos, lacking a unified framework to handle mixed-content pipelines.
• Imprecise Visual Reference Utilization: Existing diffusion methods often rely on global feature extraction, failing to capture local color distributions, textures, and lighting details from reference images. This leads to style deviations and poor instance-level consistency.
• Detail Degradation & Temporal Inconsistency:
  • Detail Loss: Standard Variational Autoencoders (VAEs) in diffusion pipelines introduce compression artifacts, erasing high-frequency physical details (e.g., metallic reflections, fabric textures).
  • Motion Artifacts: Video colorization methods often fail to model motion-aware dependencies, resulting in flickering, misalignment, and temporal incoherence during large-motion scenes.

2. Methodology: Uni-Animator

The authors propose Uni-Animator, a unified framework based on the Diffusion Transformer (DiT) architecture (initialized from Wan2.1). It accepts sketch sequences, text prompts, and visual references to generate colorized outputs. The core architecture integrates three novel technical modules:

A. Visual Reference Enhancement (Instance Patch Embedding)

To solve the issue of imprecise color transfer, the authors introduce a two-stage multi-granularity fusion strategy:

1. Coarse-Grained Fusion: Reference images are encoded via a pre-trained VAE to extract global style and structural information. This is concatenated with the initial noise latent space to guide the overall color distribution.
2. Fine-Grained Fusion (Instance Patch Embedding): To capture local details, the method partitions reference images into patches and uses a convolutional Instance Embedding module. This extracts fine-grained, instance-level features (textures, edge semantics) which are fused into the token sequence. This allows the model to align specific instances in the sketch with specific regions in the reference, supporting multiple references simultaneously.

B. Physical Detail Reinforcement

To counteract the loss of high-frequency details caused by VAE compression:

• DINO-based Feature Extraction: The framework leverages a pre-trained DINOv2 model to extract rich, structure-aware physical features (encoding material properties, surface textures, and lighting) from the visual references.
• Feature Injection: These physical features are projected and concatenated with the fused latent features.
• Cross-Modal Alignment: A cross-attention mechanism aligns the physical features with text prompts and visual references (via CLIP and T5 encoders), resolving semantic conflicts (e.g., when text says "red hair" but the reference shows blue) and ensuring the model preserves edge sharpness and physical realism.

C. Sketch-Based Dynamic RoPE

To address temporal inconsistency and flickering in dynamic scenes:

• Motion-Aware Encoding: Instead of static positional encoding, the authors propose a Dynamic Rotary Positional Embedding (RoPE).
• Mechanism: Optical flow is extracted from the input sketch sequence using RAFT. The motion is decomposed into horizontal ($u$) and vertical ($v$) components.
• Adaptive Frequency: The RoPE frequencies are dynamically scaled based on motion intensity ($\hat{M}$).
  • High Motion: Regions with rapid movement receive higher frequency parameters to capture dynamic details and prevent temporal aliasing.
  • Static Regions: Lower frequencies are maintained to ensure stability.
  • Dimension-Specific: The adaptation is applied separately to temporal, height, and width dimensions to handle asymmetric motion patterns common in animation (e.g., fast horizontal panning vs. slow vertical movement).

3. Key Contributions

1. Unified Framework: The first diffusion-based framework capable of handling both image and video sketch colorization within a single model, eliminating the need for task-specific adaptation.
2. Visual Reference Enhancement: A novel Instance Patch Embedding mechanism that enables precise, instance-level color alignment and supports multiple visual references simultaneously.
3. Physical Detail Reinforcement: A strategy using DINO features to recover and preserve high-frequency physical details often lost in standard diffusion pipelines.
4. Sketch-Based Dynamic RoPE: A motion-aware positional encoding strategy that adaptively models spatial-temporal dependencies, significantly reducing flickering and misalignment in large-motion video scenes.

4. Experimental Results

The authors evaluated Uni-Animator on a custom dataset comprising 5,000 anime clips and 30,000 filtered clips from Sakuga-42M.

• Quantitative Performance:
  • Video Colorization: Uni-Animator achieved State-of-the-Art (SOTA) performance, ranking first in FID (119.514) and LPIPS (0.193), significantly outperforming specialized video models like ToonComposer and Anidoc.
  • Image Colorization: It achieved top-tier results, outperforming methods like MagicColor and MangaNinja in SSIM, CLIP Score, and FID.
  • Metrics: The model demonstrated superior performance in temporal consistency scores and style/content alignment (CLIP Score).
• Qualitative Performance:
  • Detail Fidelity: The method successfully preserved fine textures (hair strands, fabric) and edge sharpness, which were often blurred in baselines.
  • Temporal Coherence: In video tests, the model maintained stable character identity and smooth motion without flickering, even in complex scenes with large movements.
  • Multi-Reference Handling: Unlike baselines that struggle with multiple references, Uni-Animator correctly mapped colors to specific instances across frames.

5. Significance

Uni-Animator represents a significant leap forward in automated content creation for animation and film restoration.

• Industrial Impact: By unifying image and video tasks, it drastically reduces the labor costs associated with manual frame-by-frame colorization and cross-domain adaptation.
• Technical Advancement: It demonstrates that DiT-based architectures, when combined with motion-aware encoding and physical feature reinforcement, can overcome the inherent limitations of VAEs and static positional encodings.
• Versatility: The framework's ability to handle single or multiple references with high fidelity makes it a robust tool for diverse production pipelines, from character design to full-scale animation inbetweening.

In conclusion, Uni-Animator bridges the gap between static and dynamic sketch colorization, offering a solution that is not only technically superior in detail and consistency but also practically viable for high-volume industrial workflows.