Teaching an Agent to Sketch One Part at a Time

This paper presents a method for generating controllable and editable text-to-vector sketches by training a multi-modal language model agent on a new part-level annotated dataset called ControlSketch-Part using a multi-turn process-reward reinforcement learning approach.

The Gist

Imagine you want to teach a robot to draw a picture of a chair, but instead of handing it a blank canvas and saying "Draw a chair," you want to guide it like a human teacher: "First, draw the legs. Good. Now, draw the seat. Perfect. Finally, add the backrest."

That is exactly what this paper, "Teaching an Agent to Sketch One Part at a Time," is about. The researchers built an AI agent that doesn't just spit out a finished drawing all at once; it learns to build a sketch piece by piece, just like a human artist.

Here is the breakdown of how they did it, using some everyday analogies:

1. The Problem: The "All-or-Nothing" Artist

Most AI drawing tools today work like a magic 8-ball. You ask for a "chair," and poof, a full image appears. If the legs look weird, you have to ask the whole thing to start over. It's like ordering a pizza and hoping the chef gets the pepperoni right; if they don't, you can't just ask them to move the pepperoni—you have to order a whole new pizza.

Existing methods that try to draw step-by-step are often limited. They might draw simple stick figures or icons, but they struggle to create detailed, professional-looking vector art (the kind of crisp, scalable graphics used in design software).

2. The Solution: The "Part-by-Part" Apprentice

The authors created a new AI agent that acts like a skilled apprentice. Instead of guessing the whole picture, it follows a recipe:

1. Read the instruction: "Draw a chair."
2. Break it down: "Okay, I need legs, a seat, and a back."
3. Draw one part: It draws the legs.
4. Look at the canvas: It sees the legs it just drew.
5. Draw the next part: It draws the seat, making sure it connects to the legs.
6. Repeat: It keeps going until the chair is done.

This allows for local editing. If the seat looks wrong, you can tell the AI, "Erase the seat and try a different shape," without destroying the legs you already liked.

3. The Secret Sauce: A New "Textbook" (ControlSketch-Part)

To teach the AI this skill, they needed a massive textbook. But there was a problem: no one had a dataset of professional sketches broken down into labeled parts (e.g., "these lines are the legs," "these lines are the seat").

The Analogy: Imagine trying to teach a student to write an essay, but you only have a pile of finished essays with no outlines or chapter breaks. It's hard to learn how to write.

The Fix: The team built a smart annotation pipeline. They used a powerful AI (a Vision-Language Model) to look at existing sketches and automatically:

• Deconstruct the image into parts (like separating a car into wheels, doors, and windows).
• Label the lines (paths) that make up each part.
• Critique its own work to make sure the labels are accurate.

They did this for thousands of sketches, creating a new dataset called ControlSketch-Part. It's like turning a pile of finished essays into a library of essays with detailed outlines and highlighted sections.

4. The Training: "Practice" and "Coaching"

Training the AI happened in two distinct phases, similar to how a human learns a sport:

• Phase 1: Supervised Fine-Tuning (The Drill Sergeant)
  The AI is shown the "textbook" (the ControlSketch-Part dataset). It learns the rules: "When I see the word 'legs', I must output these specific lines." It learns the format and how to draw one part at a time. This is like memorizing the rules of the game.

• Phase 2: Reinforcement Learning with Process Rewards (The Coach)
  This is the innovative part. In the first phase, the AI only saw "perfect" intermediate steps (like looking at a finished puzzle and seeing the pieces already placed correctly). But in real life, the AI has to draw the first piece, then the second, and it might make a mistake early on.

  To fix this, the researchers used a technique called GRPO (Group Relative Policy Optimization).

  • The Analogy: Imagine a coach watching a player practice. Instead of only grading the player at the end of the game, the coach gives feedback during the game. "Good pass! But your footwork on that second step was sloppy."
  • How it works: The AI draws a sketch in multiple steps. After every single step (every part added), the system checks: "Does this partial drawing look like the real thing so far?" If yes, it gets a reward. If the drawing starts to look weird, it gets a penalty. This "dense feedback" teaches the AI to correct itself as it goes, rather than waiting until the end to realize it failed.

5. The Result: A Controllable, Creative Artist

The results are impressive. The new agent:

• Draws better: It creates complex, smooth, and realistic vector sketches that look much more professional than previous methods.
• Listens better: It follows text instructions more accurately.
• Is editable: You can ask it to "change the backrest to be round" or "remove the arms," and it will do exactly that without messing up the rest of the drawing.

Summary

In short, the authors realized that to teach an AI to draw like a human, you can't just show it the final picture. You have to teach it how to build the picture, one brick at a time. They built a new "textbook" of broken-down sketches and a "coaching system" that gives feedback at every single step. The result is an AI that doesn't just generate art; it constructs it, allowing for a level of control and creativity that was previously impossible.

Technical Summary

1. Problem Statement

Current text-to-vector sketch generation methods face three primary limitations:

• Lack of Progressive Control: Most existing models generate the entire sketch in a single pass ("one-shot"), ignoring the natural, step-by-step nature of human sketching. This makes it difficult to correct localized errors or explore alternative designs for specific components without regenerating the whole image.
• Poor Editability: Without semantic grouping of strokes into meaningful "parts," editing a sketch requires modifying individual vector paths, which is inefficient and unintuitive.
• Data Scarcity: High-quality, part-annotated vector sketch datasets are rare. Collecting them manually is costly and difficult to scale, hindering the training of agents capable of iterative, part-by-part generation.

Existing solutions like SketchAgent rely on closed-source models and produce simplistic, icon-style outputs, failing to support complex, free-text guided, high-fidelity vector generation with fine-grained control.

2. Methodology

The authors propose a comprehensive framework involving a new dataset, an automated annotation pipeline, and a novel two-stage training strategy for a Vision-Language Model (VLM) agent.

A. Automated Part Annotation Pipeline

To overcome data scarcity, the authors developed a generic, scalable pipeline to annotate vector sketches (SVG) with semantic part information. The pipeline uses a VLM (Gemini 3.0 Pro) in a multi-stage loop:

1. Initial Decomposition: The VLM renders the SVG and proposes a semantic decomposition of the object into 2–5 distinct parts.
2. Critique & Refinement: The VLM acts as a critic to audit the decomposition against constraints, identifying violations and suggesting fixes.
3. Path Assignment: The VLM assigns every SVG path to a specific semantic part.
4. Diagnostic Visualization: A unique step where the VLM is shown a side-by-side comparison of the part descriptions (color-coded) and the sketch (paths colored by assigned part) to verify and correct assignment errors.
5. Caption Generation: A final short caption is generated based on the refined parts.

This process yields the ControlSketch-Part dataset, derived from the existing ControlSketch dataset, enriched with part descriptions, path-to-part assignments, and global captions.

B. Training Framework: SFT + Multi-Turn Process-Reward RL

The training of the VLM agent (based on Qwen3-VL) occurs in two stages:

1. Supervised Fine-Tuning (SFT):

   • The model learns the output format (cubic Bézier curve coordinates) and the policy for generating a single part conditioned on the current canvas and text instructions.
   • Data augmentation involves random permutations of part orders to teach the agent to handle arbitrary sequences.

2. Reinforcement Learning (RL) with Multi-Turn Process-Reward GRPO:

   • Algorithm: The authors adapt Group Relative Policy Optimization (GRPO), which eliminates the need for a separate value function model by using a group of sampled trajectories to compute a baseline.
   • Process-Reward Mechanism: Unlike standard RL that only rewards the final outcome, this method provides intermediate rewards at every step (every part added).
   • Reward Functions:
     • DreamSim Reward: Measures visual similarity between the current partial rendering and the ground-truth partial rendering using a learned perceptual metric (DreamSim).
     • Path Count Reward: Penalizes deviations in the total number of paths to prevent overly long or short trajectories.
   • Objective: This "dense credit assignment" bridges the gap between training (where the model sees ground-truth intermediate states) and inference (where it sees its own generated states), ensuring visual quality remains high throughout the generation process.

3. Key Contributions

• ControlSketch-Part Dataset: A new, high-quality dataset containing 35,000 vector sketches with rich part-level annotations (semantic descriptions and path-to-part mappings) generated via a novel automated pipeline.
• Automated Annotation Pipeline: A generic, VLM-driven pipeline capable of decomposing any vector sketch into semantic parts and assigning paths, featuring a unique diagnostic visualization step for error correction.
• Multi-Turn Process-Reward GRPO: A novel training algorithm that applies intermediate-state rewards during multi-turn generation. This enables the agent to learn progressive sketching with high visual fidelity and corrects the distribution shift between training and inference.
• Controllable Generation: The resulting agent supports free-text guided, part-by-part generation and localized editing (removal/replacement of specific parts) without regenerating the entire sketch.

4. Results

The method was evaluated against state-of-the-art baselines including SketchAgent, Gemini 3.1 Pro, and a SDXL + SwiftSketch pipeline.

• Quantitative Metrics: The proposed model (SFT + RL) achieved the highest Long-CLIP cosine similarity scores, indicating the generated sketches are most faithful to the text prompts. It outperformed the SFT-only variant, proving the efficacy of the RL stage.
• User Studies: In double-blind preference tests, human evaluators consistently preferred the authors' method over all baselines for both step-by-step generation quality (matching part descriptions) and final visual quality.
• Qualitative Analysis: The model produces sketches with smooth paths, natural styles, and identifiable parts. It successfully handles complex objects (e.g., animals, robots) where baselines often fail with distorted components or simple geometric primitives.
• Ablation Studies: Experiments confirmed that both the multi-turn formulation and the process-reward mechanism (vs. outcome-only rewards) are critical for performance.

5. Significance

This work represents a significant shift in text-to-vector synthesis by moving away from holistic, one-shot generation toward iterative, part-based creation.

• Workflow Integration: By mimicking human sketching workflows (drawing, reviewing, modifying), the tool is more practical for creative professionals who require fine-grained control and the ability to branch off ideas at intermediate stages.
• Data Efficiency: The automated annotation pipeline demonstrates that high-quality, structured training data for complex vector tasks can be synthesized at scale, reducing reliance on expensive manual labeling.
• Future Direction: The success of intermediate visual rewards suggests a promising direction for training agents in other sequential visual generation tasks where maintaining consistency over long horizons is challenging.