VideoSketcher: Video Models Prior Enable Versatile Sequential Sketch Generation

VideoSketcher introduces a data-efficient, two-stage approach that adapts pretrained text-to-video diffusion models to generate high-quality, temporally coherent sequential sketches by leveraging LLMs for stroke ordering and distilling visual appearance from minimal human-drawn examples.

The Gist

Imagine you want to teach a robot how to draw. Most current AI models are like photographers: they look at a finished painting and try to guess what it looks like, or they spit out a final image instantly. They don't really understand how the artist moved their hand to create it.

VideoSketcher is different. It's like teaching the robot to be a storyteller who draws. Instead of just showing you the final picture, it shows you the entire movie of the drawing process, stroke by stroke, exactly how a human would do it.

Here is the simple breakdown of how they did it, using some everyday analogies:

1. The Problem: The "Two-Headed" Robot

The researchers realized that to draw well, you need two different brain powers:

• The Planner: You need to know what to draw and in what order (e.g., "Draw the body first, then the head, then the tail").
• The Artist: You need to know how to make the lines look smooth, wobbly, and realistic.

Previous AI models were bad at one or the other:

• Text-only AI (LLMs): Great at planning ("Draw a cat"), but their drawings looked like a child's scribbles. They knew the order but had bad "motor skills."
• Image/Video AI: Great at making things look pretty, but they didn't know the rules of drawing. If you asked them to draw a cat, they might draw the tail before the body, or draw the whole cat instantly like a magic trick.

2. The Solution: The "Video Director"

The team realized that Video AI models (which are trained on millions of hours of movies) are actually perfect for this. Why? Because movies are all about time and movement.

They treated a sketch not as a static picture, but as a short movie where black lines slowly appear on a white screen.

• The Analogy: Imagine a time-lapse video of someone painting a wall. You see the roller move, the paint spread, and the wall fill up. VideoSketcher treats a sketch exactly like that time-lapse video.

3. The Secret Sauce: The "Two-Stage Training"

You can't just throw a robot into an art class and expect it to learn in one day. The researchers used a clever two-step training camp:

Stage 1: Learning the "Grammar" of Drawing (The Lego Phase)

• They didn't start with complex cats or cars. They started with simple shapes: circles, squares, and triangles.
• The Analogy: Think of this like teaching a child to write letters before writing a novel. They taught the AI to draw a circle, then a square, then a triangle, strictly following a rule like "Draw the circle first, then the square."
• This taught the AI the rules of order without getting confused by fancy details.

Stage 2: Learning the "Style" (The Art Class Phase)

• Once the AI knew how to order the strokes, they showed it just seven real human drawings (a lamp, a car, a tree, etc.).
• The Analogy: This is like giving the robot a single sketchbook from a master artist. The robot already knows the rules of order from Stage 1, so it just needs to learn the "handwriting" style.
• Because it already understood the process, it only needed a tiny amount of human data to learn the look.

4. What Can It Do? (The Magic Tricks)

Because they built this on video technology, the robot can do things other drawing AIs can't:

• The "Co-Draw" Feature: You can draw a line, and the robot will guess what you are making and finish the rest of the drawing for you. Then you can change a line, and it adapts. It's like a collaborative dance where you and the robot take turns leading.
• The "Magic Brush": You can show the robot a picture of a specific paintbrush or a marker color, and it will use that exact style for the whole drawing. It's like handing the robot a specific pen and saying, "Use this one."
• The "Director's Cut": You can tell the robot, "Draw the body first, then the head," or "Draw the head first, then the body." The robot will follow your specific instructions and change the order of the drawing movie accordingly.

Why Does This Matter?

Most AI today feels like a black box: you type a prompt, and a picture pops out. You have no control over the process.

VideoSketcher opens the box. It turns drawing into a conversation. It understands that drawing is a journey, not just a destination. By using video models, it learned that the way you draw something is just as important as the final result, allowing for a much more natural, creative, and interactive experience between humans and machines.

In short: They taught a robot to draw by showing it movies of people drawing, first with simple shapes to learn the rules, and then with a few real examples to learn the style. Now, it can draw with you, not just for you.

Technical Summary

1. Problem Statement

Sketching is inherently a sequential process where strokes are drawn in a meaningful order to explore and refine ideas. However, existing generative models face significant limitations in replicating this process:

• Static Image Models: Most generative models treat sketches as static images, ignoring the temporal structure and stroke ordering essential to creative drawing.
• Data-Heavy Autoregressive Models: Early approaches like SketchRNN rely on millions of human-drawn sequences, restricting them to fixed object categories and limiting stylistic diversity.
• LLM Limitations: Recent methods like SketchAgent use Large Language Models (LLMs) to plan stroke order via text commands. While they excel at semantic planning, they lack strong visual priors, resulting in sketches that are semantically coherent but visually simplistic, child-like, and low-quality.
• Video Reconstruction vs. Generation: Existing video models (e.g., PaintsUndo) focus on reversing the creation process of an existing image, rather than generating new sketches from text with explicit, stroke-by-stroke progression.

The core challenge is to generate high-quality, sequential sketch videos that follow natural stroke ordering and exhibit rich visual detail without requiring massive datasets of human-drawn sketches.

2. Methodology

The authors propose VideoSketcher, a data-efficient approach that adapts pretrained text-to-video diffusion models to generate sketching processes. The method leverages the complementary strengths of LLMs (semantic planning) and video diffusion models (high-quality visual rendering).

Key Components:

A. Sketch Representation

• Sketches are represented as short videos in pixel space, where black strokes are progressively drawn on a blank canvas.
• Training data is constructed by parsing SVGs (Scalable Vector Graphics) recorded by artists. These are rendered into videos where strokes appear gradually along their path, ensuring clean temporal structure (one stroke per frame or continuous stroke motion).

B. Two-Stage Fine-Tuning Strategy
To decouple the learning of stroke ordering from visual appearance, the authors employ a two-stage fine-tuning process on a pretrained video diffusion model (Wan 2.1):

1. Stage 1: Learning the "Grammar" of Shape Compositions

   • Goal: Teach the model to follow text-specified stroke ordering.
   • Data: A synthetic dataset of simple geometric primitives (circles, rectangles, curves) arranged in spatial configurations based on Gestalt principles (containment, adjacency, overlap).
   • Mechanism: The model learns to associate linguistic ordering cues (e.g., "first draw X, then Y") with the temporal sequence of stroke additions, independent of complex visual styles.

2. Stage 2: Distilling Sketch Appearance

   • Goal: Adapt the model to the visual style of hand-drawn sketches.
   • Data: A very small dataset of only seven manually authored sketching processes (lamp, car, chair, tree, cup, butterfly, flower).
   • Mechanism: Since the model already learned ordering in Stage 1, this stage transfers the visual aesthetic and abstraction level to the target domain without relearning the ordering logic.

C. Extensions

• Brush Conditioning: The model supports image-to-video generation where a user provides a brush exemplar (style/color) in the first frame. The model generalizes this style to unseen brushes and colors.
• Autoregressive Generation: To enable real-time interaction, the authors use the diffusion-based model to synthesize a larger dataset, which is then used to fine-tune an autoregressive video model (CausVid). This allows for turn-based human-model co-drawing.

3. Key Contributions

1. Novel Paradigm: Introduces the first method to use pretrained video diffusion models as priors for sequential sketch generation, bypassing the need for massive sketch-specific datasets.
2. Data Efficiency: Demonstrates that high-quality sequential sketches can be generated with only seven human-drawn examples by decoupling ordering learning (via synthetic data) from appearance learning.
3. Complementary Architecture: Successfully combines LLMs for semantic planning (stroke order) with video diffusion models for high-fidelity rendering, overcoming the visual limitations of pure LLM-based sketching.
4. Versatile Control: Enables brush style conditioning (controlling stroke texture/color via visual prompts) and interactive co-drawing via autoregressive adaptation.
5. Temporal Coherence: Achieves semantically meaningful stroke ordering (e.g., drawing the body before the head) that closely mimics human drawing behavior, unlike baselines that collapse temporal progression.

4. Results

The method was evaluated against baselines including SketchAgent, PaintsUndo, and human drawings from the QuickDraw dataset.

• Semantic Recognition: On a CLIP-based zero-shot classification task (50 categories), VideoSketcher achieved 82% Top-1 accuracy, significantly outperforming SketchAgent (48%) and human QuickDraw sketches (52%). It also reached 96% Top-5 accuracy.
• Temporal Progression: Analysis of the drawing process showed that VideoSketcher's semantic recognizability increases gradually over time, mirroring human drawing. In contrast, PaintsUndo saturates immediately (reconstructing details upfront), and SketchAgent often fails to convey recognizable concepts until the end.
• Brush Style Generalization: The model successfully transferred brush styles and colors not seen during training, reducing the Gram matrix distance to the target style by 49% compared to a random baseline.
• Ablation Studies:
  • Models trained only on primitives had good ordering but poor aesthetics.
  • Models trained only on human sketches had good aesthetics but poor ordering.
  • The Two-Stage model combined both strengths, achieving the best balance of ordering fidelity and visual quality.
• Autoregressive Performance: The AR version achieved 45% Top-1 accuracy (comparable to human/LLM baselines) and enabled real-time, turn-based co-drawing, though with slightly lower visual fidelity than the diffusion-based version.

5. Significance and Impact

• New Perspective on Creative AI: The paper shifts the focus from generating static images to modeling the creative process itself. It proves that video models, traditionally trained on photorealistic content, can be effectively distilled to understand abstract, sequential drawing behaviors.
• Democratization of Sketch Generation: By requiring only a handful of training examples, the method lowers the barrier for creating specialized sketching models, avoiding the need for massive, curated datasets of human drawings.
• Interactive Collaboration: The ability to perform human-model co-drawing in real-time opens new avenues for visual brainstorming, educational tools, and collaborative prototyping, where the AI acts as a partner that understands both what to draw and how to draw it step-by-step.
• Limitations: The paper acknowledges that pixel-space generation can occasionally produce multiple strokes in a single frame (violating strict stroke continuity) and that the model relies on the video model's existing knowledge base (struggling with highly specialized concepts like complex math functions compared to LLMs).

In conclusion, VideoSketcher establishes a powerful framework for sequential sketch generation, leveraging the temporal priors of video diffusion models to create high-quality, ordered, and interactive drawing experiences with minimal data.