VectorGym: A Multitask Benchmark for SVG Code Generation, Sketching, and Editing

1. Problem Statement

Scalable Vector Graphics (SVG) are widely used in web design and digital media because they function as executable code (exposing geometry, style, and structure) rather than static raster images. Despite rapid advancements in Vision-Language Models (VLMs), the field lacks a rigorous, realistic benchmark for SVG generation and manipulation.

Key limitations of existing approaches include:

• Synthetic Data: Prior benchmarks often rely on synthetic, programmatic edits or simple icons, failing to capture the complexity of real-world design workflows.
• Limited Scope: Existing datasets rarely cover the full spectrum of tasks, such as converting rough hand-drawn sketches to code, performing complex multi-step edits, or generating detailed captions from code.
• Lack of Human Ground Truth: Most datasets lack "gold-standard" human annotations that capture semantic intent, design logic, and complex primitive usage (e.g., gradients, text, animations).
• Evaluation Gaps: Traditional metrics (pixel MSE, CLIP) fail to evaluate the structural validity, semantic correctness, and instruction-following capabilities required for vector code generation.

2. Methodology

A. The VectorGym Benchmark

The authors introduce VectorGym, a comprehensive suite of four tasks built from diverse, real-world SVGs sourced from GitHub (via the SVG-Stack dataset). The benchmark features expert human annotations for all tasks, ensuring high-quality ground truth.

The four core tasks are:

1. VG-Sketch (Sketch2SVG): Converting rough, hand-drawn bitmap sketches (with imperfect lines) into clean, scalable SVG code. This tests spatial reasoning and abstraction.
2. VG-Edit (SVG Editing): Applying natural language instructions to existing SVGs to perform complex, multi-step edits (e.g., changing chart types, modifying text, adding new primitives). This tests structural understanding and semantic manipulation.
3. VG-Text (Text2SVG): Generating complete SVG code from natural language descriptions, ranging from simple shapes to complex illustrations.
4. VG-Cap (SVG Captioning): The inverse of Text2SVG; generating accurate natural language descriptions from SVG code, testing code comprehension and visual understanding.

Dataset Construction:

• Source: 7,000 candidate samples filtered from SVG-Stack (icons, diagrams, fonts, emojis).
• Filtering: Applied token length constraints (2k–8k tokens), color entropy thresholding, and human visual inspection.
• Annotation: Over 20 expert annotators (designers, coders) created sketches, complex edits, and detailed captions. Edits were specifically designed to be non-trivial, requiring semantic understanding (e.g., "replace the sphere with a triangle and change the text").

B. Evaluation Metrics: VLM-as-a-Judge

Recognizing that traditional metrics (MSE, LPIPS) are insufficient for vector code, the authors propose a VLM-as-a-Judge (VLMAJ) metric.

• Method: A suite of VLMs evaluates generated outputs against ground truth based on specific rubrics (visual fidelity, semantic alignment, code validity).
• Validation: The authors conducted correlation studies between VLM scores and human expert ratings. Gemini 3 Pro was identified as the best judge for Sketch, Caption, and Text tasks, while GPT-5.1 was best for Editing.
• Overall Score: A unified score is computed by averaging task-specific scores (VLM Judge, DINO, inverted MSE/LPIPS, ROUGE/BGE-M3), all scaled to [0, 100].

C. Training Method: Multi-Task Reinforcement Learning

The authors propose a reinforcement learning approach to train a Qwen3-VL 8B model on all four tasks simultaneously.

• Framework: Reinforcement Learning from Rendering Feedback (RLRF) using GRPO (Group Relative Policy Optimization).
• Reward Signal:
  • For Generation/Editing (Sketch, Text, Edit): The model outputs SVG code, which is rendered to raster images. Rewards are calculated based on perceptual similarity (LPIPS, DINO) and pixel distance (MSE) between the rendered prediction and the ground truth.
  • For Captioning: Rewards are based on embedding similarity (BGE-M3) between the predicted and ground-truth text.
• Curriculum Learning: To improve stability, samples are sorted by response length (proxy for difficulty) within each task, allowing the model to progress from simple to complex examples.
• Efficiency: The model is trained on a single node (8x H200 GPUs) for 600 iterations (~2 days), achieving state-of-the-art results among open-source models.

3. Key Contributions

1. VectorGym Benchmark: The first multi-task benchmark for SVG covering Sketch2SVG, complex Editing, Text2SVG, and Captioning, featuring gold-standard human annotations and real-world "in-the-wild" data.
2. Novel Tasks: Introduction of the Sketch2SVG task and the first dataset of complex, human-authored SVG edits that require semantic reasoning and multi-step transformations.
3. RL Training Strategy: A multi-task RL method using rendering-based rewards and curriculum learning that enables a small 8B model to outperform much larger proprietary and open-source models.
4. VLM-as-a-Judge: A validated evaluation protocol for SVG generation that correlates strongly with human judgment, addressing the lack of reliable metrics for vector code.

4. Results

The authors evaluated frontier VLMs (both proprietary and open-source) on VectorGym.

• Performance Hierarchy:
  • Proprietary Leaders: Gemini 3 Pro achieved the highest overall score (73.17), followed by GPT-5.1 (71.36).
  • Open-Source Breakthrough: The authors' fine-tuned Qwen3-VL 8B Gym model achieved an overall score of 66.05.
    • It surpassed GPT-4o (64.93).
    • It significantly outperformed the much larger Qwen3-VL 235B (62.32).
    • It matched or exceeded GPT-4o in specific tasks like Sketch2SVG (70.72 vs 69.55) and SVG Editing (82.81 vs 82.35).
• Task Difficulty: The benchmarks revealed a clear difficulty hierarchy:
  1. Text2SVG (Easiest)
  2. SVG Editing
  3. Sketch2SVG
  4. SVG Captioning (Hardest)
• Qualitative Analysis: The RL-trained 8B model demonstrated superior ability to follow complex multi-step editing instructions and preserve topological structures in sketches compared to base models and larger baselines.

5. Significance

• Advancing Visual Code Generation: VectorGym establishes a rigorous standard for evaluating models on "visual-to-code" tasks, moving beyond simple image generation to structured, editable vector graphics.
• Efficiency vs. Scale: The results demonstrate that specialized, smaller models trained with high-quality, task-specific RL and curriculum learning can outperform massive general-purpose models in niche domains like SVG generation.
• Real-World Applicability: By focusing on complex, human-authored edits and sketches, the benchmark aligns closely with professional design workflows, providing a realistic testbed for AI tools intended to assist designers.
• Open Science: The benchmark and the trained models are publicly available on HuggingFace, fostering reproducibility and further research in vector graphics generation.

In conclusion, VectorGym fills a critical gap in the AI research landscape by providing a challenging, human-validated benchmark for SVG manipulation, proving that targeted reinforcement learning can unlock state-of-the-art performance in visual code generation even with parameter-efficient models.