🤖 machine learning

Draw This First

This paper introduces a novel sketch generation approach that inverts the typical stroke-ordering process by predicting a 2D field to define drawing sequences, enabling the creation of ordered vector sketches from text or images that adhere to specific drawing instructions.

Original authors: Dazhi Zhong, Rowan Bradbury, Grant Davis

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dazhi Zhong, Rowan Bradbury, Grant Davis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician draw a picture on a canvas. Usually, when we talk about computers drawing, we think of them as artists who just spit out the final picture all at once, or maybe they build it up pixel by pixel like a slow-motion time-lapse. But there's a different way to think about drawing: the story of how the lines are made. Did the artist draw the outline first, then the eyes, then the hat? Or did they start with the hat, then the eyes, and finish with the outline? This "story" is called the drawing order.

For a long time, computers were great at making the final picture look right, but they were terrible at figuring out the story of how to make it. They didn't know which line came first. This matters because if you want a computer to help you draw, or if you want to turn a photo back into a simple sketch that a robot can actually trace with a pen, you need to know the order. It's the difference between handing someone a finished puzzle and handing them the instructions on how to put the pieces together. This paper lives in the world of computer vision and generative AI, where machines try to understand and create images. The big question here is: Can we teach a computer not just what to draw, but how to draw it step-by-step, just like a human?

The Paper's Big Idea: Painting with Time

The authors of this paper, working with tools from Krea.ai and Wand Technologies, decided to flip the script on how computers generate sketches. Instead of trying to predict the next line in a sequence (which is like trying to guess the next word in a sentence one by one), they asked a different question: What if we could paint a map that tells the computer exactly when to draw every single line?

Think of it like a "heat map" or a "traffic light system" for a drawing. Imagine you have a blank piece of paper. The computer doesn't just draw a cat; it first paints a invisible, colorful layer over the paper. On this layer, the color of every pixel tells the computer how "old" that part of the drawing is.

Red pixels might mean "Draw this part first."
Yellow pixels mean "Draw this part next."
Blue pixels mean "Draw this part last."

This is the core trick: they turned time (the order of drawing) into color.

How They Taught the Computer

To make this work, the team needed a teacher. They didn't use the usual messy "doodle" datasets found on the internet. Instead, they hired 50 professional artists to draw 47,318 high-quality sketches. These artists drew everything from birds to teapots, and the computer recorded every single stroke they made, exactly as they made it.

The computer then looked at these drawings and created its special "time-as-color" map. It took the real drawing and turned it into a weird, colorful image where the brightness and hue of the ink told the story of the drawing process.

Then, they used a powerful AI model (a "diffusion transformer," which is a type of AI that learns by guessing and correcting itself) to learn how to create these colorful maps from scratch. They taught the AI two things:

Text-to-Sketch: If you type "a cat," the AI should generate the colorful map that tells you how to draw a cat.
Image-to-Sketch: If you show the AI a photo of a cat, it should figure out the drawing order and generate the map.

The Magic Trick: Reading the Map

Once the AI generates this colorful "time map," the computer has to turn it back into a real drawing. This is where their special decoder comes in. It looks at the colors, figures out which pixels are red (first), which are yellow (second), and so on. It then connects the dots to create a smooth, ordered line drawing.

The coolest part? You can tell the AI to change the story. If you type, "Draw the tail first, then the body," the AI listens. It doesn't just change the picture; it changes the order in which the lines appear, even if the final picture looks the same. It's like telling a storyteller to start their story at the end and work backward, but the story still makes sense.

What They Found (and What They Didn't)

The results are pretty impressive, but the authors are careful to say exactly what works and what doesn't.

It works really well with instructions: When they told the AI to draw in a specific order (like "head, then body, then tail"), the AI followed those instructions almost perfectly. The order of the lines matched their commands much better than random guessing.
It keeps the "vibe" of the original art: Even though they changed how the AI draws, it still kept the artistic style and knowledge of the original model. It could draw things it had never seen before, just by following the order instructions.
It's not perfect yet: The authors admit that while the computer is great at following big instructions (like "draw the head first"), it sometimes gets confused about the tiny details inside a single part (like the exact order of lines inside the head). Also, the computer sometimes breaks a single smooth line into a few smaller pieces, so the final drawing has a few more lines than the original artist intended.

Why This Matters

This paper shows that we can teach computers to understand the process of drawing, not just the result. By turning time into color, they found a clever way to make AI follow instructions about how to create art. It's a step toward computers that don't just mimic human art, but understand the human way of thinking about it—one stroke at a time.

The authors suggest that this method could help in turning photos into editable vector sketches or helping robots learn to draw, but they stop short of claiming it's a solved problem. They show that with the right training and a little bit of "color-coded time," machines can start to draw like humans, one step at a time.

Technical Summary: Draw This First

Problem Statement

The field of sketch generation is diverse, with existing methods typically treating the drawing order as either an intrinsic output of the model (via autoregression or diffusion) or an unrepresented factor. Current approaches often fail to expose drawing order as a free-form, language-conditioned variable. Furthermore, many systems rely on low-quality "doodle" corpora, and few can generate vector sketches that strictly adhere to user-specified instructions regarding the sequence of strokes. The challenge lies in creating a system that can generate ordered vector strokes from text or images, allowing the user to explicitly dictate the drawing sequence (e.g., "draw the head first, then the body") without compromising the geometric fidelity of the sketch.

Methodology

The authors propose a novel formulation that inverts the typical sketch generation process: instead of predicting strokes sequentially, the model predicts a 2D field that defines the order in which strokes are drawn. The pipeline consists of three main components:

1. Dataset and Annotation

The model is trained on a commissioned corpus of 47,318 high-quality handmade artist drawings, significantly exceeding the quality of existing doodle datasets.

Hierarchical Annotation: Each drawing is annotated with a two-level bounding-box hierarchy (region and subject).
Order Captions: The authors synthesize order-descriptive captions that specify the sequence of visits to named regions (e.g., "box1 left, then box2, then box1 right, revisited").
Permutation Strategy: To teach the model to follow arbitrary orders, the target stroke order is permuted while preserving the natural order within regions (rough outline to details). The captions are programmatically regenerated to match the permuted order, creating order-supervised pairs. This deliberately decorrelates order from geometry during training.

2. Order-as-Color Intermediate Representation

To enable training within a pretrained image latent space, the authors construct an image-native intermediate representation using an order-as-color codec:

Global Arc ( $a$ ): Represents the total drawing progress from 0 to 1.
Within-Stroke Arc ( $u$ ): Represents progress within a specific stroke.
Encoding: The sketch is encoded into an HSV image where:
- Hue ( $H$ ): Encodes the global arc ( $a \cdot 342/360$ ).
- Saturation ( $S$ ): Encodes the foreground mask ($1$).
- Value ( $V$ ): Encodes the within-stroke arc ( $1 - u/2$ ).
This results in a rasterized "codec image" where the drawing order is visually encoded as a color sweep.

3. Model Architecture

The system utilizes a pretrained latent flow-matching transformer (based on Qwen-Image-Edit-2509) and a custom decoder:

Generator (DiT): A Diffusion Transformer adapted with LoRA (Rank-64) is trained to generate the latent representation of the codec image from text or image conditions.
Order-Native Decoder: A finetuned VAE decoder reads the generated latent and emits three targets:
1. Global Arc Field ( $a'$ ): The predicted order field.
2. Foreground Mask ( $m'$ ): The stroke mask.
3. Instance Embeddings ( $e'$ ): Eight-dimensional embeddings for stroke segmentation.
Vectorization: The decoder outputs are processed without a learned tracer. HDBSCAN clusters the embeddings to segment strokes. These clusters are sorted by the mean predicted arc field. Within each cluster, a nearest-neighbor walk extracts pixels, which are then smoothed and simplified using Ramer-Douglas-Peucker (RDP) to produce the final ordered vector polyline.

Key Contributions

Inverted Formulation: The paper introduces a method where the model predicts a 2D order field rather than a sequence of strokes, allowing order to be a controllable, language-conditioned variable.
Order-as-Color Codec: A novel encoding scheme that maps drawing progress to HSV color channels, enabling the use of pretrained image diffusion models for sequential sketch generation.
High-Quality Dataset: The creation of a 47k-image dataset of professional artist drawings with hierarchical bounding box annotations and synthesized order captions.
Instruction Following: The ability to generate sketches that follow text instructions specifying drawing orders, with adherence varying based on the granularity of the instruction (reliable at the region level, but degrading at part and pass levels).

Results

The model was evaluated on held-out data from the training set and external datasets (Creative Birds, Creative Creatures, FS-COCO, QuickDraw).

Order Accuracy:
- When provided with an order instruction, the model achieves a Kendall $\tau$ of 0.449 on in-domain data and 0.373 on out-of-domain data, significantly outperforming geometry-only baselines (0.096).
- Reversing the instruction successfully inverts the drawing order (Kendall $\tau$ of $-0.267$), demonstrating that the text, not the image, drives the sequence.
- Adherence is highly dependent on instruction granularity: region-level instructions are followed reliably ( $\tau \approx 0.84$ ), but adherence drops to 0.46 at the part level and 0.44 at the pass level.
- Control is further limited by caption structure; a single trailing constraint is often lost entirely (e.g., a named region drawn first is ignored 68.5% of the time when appended to a full caption), and reversing instructions that conflict with the description's mention order causes adherence to drop significantly.
- The "decoder ceiling" (using ground-truth latents) shows a potential Kendall $\tau$ of 0.91–0.94, indicating the bottleneck lies in the generative model's ability to predict the latent, not the vectorization process.
Geometry Preservation:
- Order interventions do not alter the geometry. Stroke match rates remain high (0.89–0.95) and mask IoU remains stable regardless of the order instruction.
- The model retains the open-vocabulary drawing capabilities of the pretrained prior, achieving Top-1 CLIP recognition scores of 0.70 on unseen QuickDraw categories.
Limitations Observed:
- Granularity: Control degrades significantly as instruction granularity increases. While region-level order is followed reliably, part-level and pass-level adherence drops to $\approx 0.45$ , and within-unit order agreement is near zero.
- Fragmentation: The vectorization process recovers paths with 1.7–2.5 $\times$ the true stroke count, resulting in fragmented strokes.
- Loss of Information: The objective discards time, pressure, tilt, and brush width, outputting 1px unweighted centerlines.
- No Intrinsic Prior: Due to the training strategy, the model has no verifiable human-like global order prior when no instruction is given; it relies on heuristics or the caption's mention order.

Significance and Claims

The paper claims to present a system that successfully derenders sketches faithfully and retains the strong text-to-sketch generation capabilities of the underlying pretrained model while adding the novel ability to follow explicit text instructions for drawing order.

The authors emphasize that this approach bridges the gap between raster-based generation and vector output, allowing for "replayable polyline paths" where the sequence is a free-form language variable. They assert that the model can follow instructions specifying arbitrary orders, even when those orders contradict the recorded artist sequence, proving that order can be decoupled from geometry and controlled via natural language. However, they remain modest regarding the limitations, acknowledging that control is constrained by the unit size used in permutation training, that adherence degrades significantly at finer granularities (part/pass level), and that trailing constraints in captions are often lost. They also note that the current output lacks the rich stroke dynamics (pressure, width) of the original input.