💻 computer science

StrokePlan-RNN: Predicting Ordered Drawing Procedures from Face Line Art

This paper introduces StrokePlan-RNN, an image-conditioned autoregressive model that successfully predicts ordered stroke sequences from static face line art by training on a newly constructed dataset, achieving high geometric accuracy while highlighting the need for improved procedural evaluation metrics.

Original authors: Gameil S. H. Ali, Entesar Altaheri

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gameil S. H. Ali, Entesar Altaheri

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a finished painting. You can see the colors, the shapes, and the final masterpiece. But have you ever wondered about the invisible dance that created it? Did the artist paint the sky first, then the mountains, or did they start with the tiny details in the foreground? For computers, seeing an image is easy; understanding the story of how it was built is a much harder puzzle. This is the world of "procedural visual prediction." While modern AI is great at creating new pictures from scratch or recognizing what's in a photo, it often treats images like static grids of pixels, missing the secret recipe of the brushstrokes. Scientists are now trying to teach computers to reverse-engineer this recipe, asking not just "what does this look like?" but "how would a human actually draw this?" If we can crack this code, we could build robots that teach us to draw, or software that turns a finished sketch into a step-by-step tutorial, revealing the hidden logic behind human creativity.

Enter StrokePlan-RNN, a new computer model designed to solve this specific mystery for face drawings. Think of it as a digital detective that looks at a completed line drawing of a face and tries to guess the exact order in which a human artist would have drawn it. The researchers didn't just ask the computer to guess; they gave it a special training manual. They created a custom dataset of 640 face line art images. For every single image, they didn't just save the final picture; they recorded the artist's hand moving in real-time, capturing every single stroke in the exact order it was made. They organized these strokes into logical groups, like "face outline," "eyes," "nose," and "hair," creating a structured timeline of how a face is built from the ground up.

The model itself works like a two-part team. First, there's an "encoder" (a ResNet-18) that looks at the finished face and creates a mental map of its structure. Then, there's a "decoder" (a two-layer LSTM) that acts like a virtual artist. This decoder starts drawing, one stroke at a time. It looks at the mental map and the stroke it just drew, then predicts the very next stroke it should make. It keeps doing this, stroke by stroke, until it decides the drawing is finished. The goal is for the computer to replay the drawing in a way that looks like a natural, step-by-step construction, starting with big shapes and filling in the details later.

The results suggest that the model is getting the hang of this. On a test set of images it had never seen before, StrokePlan-RNN managed to predict stroke paths that were off by an average of only 3.95 pixels (on a 512 × 512 pixel image). That's a very small error, meaning the lines land almost exactly where they should. The model also guessed the total number of strokes needed with an error of just 3.1 strokes on average. When the researchers compared this to a "random order" baseline (where strokes were shuffled randomly) or a version of the model that couldn't see the image at all, StrokePlan-RNN was far superior. The random baseline had a much larger error of 11.42 pixels, suggesting that knowing the correct order really does help the computer draw better.

However, the authors are careful not to claim they have solved the problem completely. While the model is excellent at drawing the lines in the right place, it is still a bit fuzzy on whether it truly understands the story of the order. The current tests measure how close the lines are to the original, but they don't directly measure if the computer drew the eyes before the nose, or the hair last, in the exact way a human teacher would. The researchers found that the model sometimes makes "local ordering errors," like drawing an eyebrow in the middle of two eye strokes when it should have waited. They also noticed it sometimes stops too early or draws too many extra lines.

The paper suggests that while the model has learned a "coarse" strategy—starting with the big outline and moving to details—it hasn't fully mastered the fine-grained, instructional logic of a human artist yet. The researchers admit that their dataset is relatively small and that the model was trained on just one specific way of drawing faces, so it might not know all the different ways humans actually draw. They propose that future work needs to focus on new ways to measure "order" specifically, rather than just how accurate the lines are. In short, StrokePlan-RNN is a promising first step that shows computers can start to see the process behind a picture, but there is still a long way to go before it can truly teach us how to draw like a human.

Technical Summary: StrokePlan-RNN

Problem Definition
The paper addresses the problem of recovering an ordered drawing procedure from a completed static image. Specifically, given a face line art image (a raster image), the goal is to infer not only the constituent vector strokes but also the temporal sequence in which they should be executed to plausibly reconstruct the image. This task, termed "procedural reconstruction," differs from standard sketch generation (which creates new images from latent codes) or handwriting synthesis (which models continuous trajectories). The core challenge is inferring the temporal structure of a finished drawing solely from its spatial structure.

Methodology
The authors propose StrokePlan-RNN, an image-conditioned autoregressive model designed to predict ordered stroke sequences.

Dataset Construction: A custom dataset of 640 face line art images was created. Unlike standard datasets, each sample includes a completed image paired with a structured, temporally ordered sequence of strokes. The annotation process utilized a custom tool to record pen trajectories, which were then organized into seven semantic facial groups (face outline, eyes, eyebrows, nose, mouth, ears, hair). Strokes were normalized by snapping points to the underlying line art and resampling to a fixed length of 32 points per stroke.
Architecture: The model follows an encoder-decoder paradigm:
- Encoder: A ResNet-18 backbone (pre-trained on ImageNet and fine-tuned) processes the input 512×512 line art image to produce a 512-dimensional visual embedding ( $z$ ).
- Decoder: A two-layer LSTM with a hidden size of 512 generates strokes autoregressively. At each time step $t$ , the decoder receives the image embedding $z$ , the previous hidden state, and a projection of the previously generated stroke $s_{t-1}$ .
- Outputs: The decoder predicts the next stroke (64 dimensions representing 32 2D points) and an End-of-Sequence (EOS) probability via a sigmoid head.
Training Objective: The model is trained end-to-end using two loss components:
1. Stroke Reconstruction Loss: Smooth L1 loss over the resampled point coordinates.
2. Stopping Loss: Binary cross-entropy for the EOS prediction.
  Training employs teacher forcing, where the ground-truth previous stroke is fed as input during training.

Key Contributions

Task Formalization: The paper formalizes the task of ordered drawing-plan prediction from static line art, distinguishing it from generative sketching or interactive guidance.
Dataset: The creation of a dataset of 640 annotated face line art images with semantically grouped, temporally ordered stroke sequences, along with the associated annotation tooling.
Baseline Model: The proposal of StrokePlan-RNN as an image-conditioned autoregressive baseline for this task.
Evaluation Analysis: An initial quantitative and qualitative assessment of the model's ability to recover procedural organization, coupled with a critical identification of the gap between geometric reconstruction metrics and true procedural ordering evaluation.

Results
On a held-out test set (64 samples), StrokePlan-RNN achieved the following performance:

Geometric Accuracy: A mean point error of 3.95 pixels on 512×512 images.
Sequence Length: A sequence length error of 3.1 strokes (approximately 6.7% of the average sequence length).
Baseline Comparisons: The model substantially outperformed two baselines:
- A Random Stroke Order baseline (shuffled ground truth), which resulted in a much higher point error (11.42 px), indicating that coherent ordering provides critical predictive signal.
- A Decoder-Only (No Encoder) ablation, which yielded a point error of 6.83 px, confirming that image conditioning is essential for spatial accuracy.

Qualitative analysis of replay sequences suggests the model captures a "coarse-to-fine" procedural organization (e.g., drawing the face outline before internal features), consistent with the annotation protocol. However, systematic failure modes were identified, including premature stopping, over-generation, local ordering errors (correct strokes in wrong temporal order), and instability in dense-detail regions (eyes, hair).

Significance and Claims
The paper positions this work as a first step toward "procedural visual understanding"—learning not just what an image depicts, but how it could be constructed. The authors claim that StrokePlan-RNN successfully learns to map image features to spatially accurate stroke paths and exploits sequential structure to improve geometric prediction.

However, the authors are modest regarding the extent of procedural learning. They explicitly state that current evaluation metrics primarily measure geometric reconstruction accuracy rather than ordering quality directly. Consequently, while qualitative replays suggest the model learns coarse procedural strategies, the paper does not claim to have fully solved the problem of recovering fine-grained, pedagogically valid instructional sequences. The authors emphasize that the model learns a specific, controlled annotation protocol rather than the full diversity of human drawing strategies, and they identify the development of order-aware evaluation metrics (such as Pairwise Order Accuracy and Kendall's Tau) as the most critical direction for future work to validate procedural competence.

More like this