💻 computer science

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

The paper introduces VFIG, a family of Vision-Language Models that converts rasterized figures into high-fidelity SVGs by leveraging a large-scale dataset (VFIG-DATA), a coarse-to-fine training curriculum combining supervised fine-tuning and reinforcement learning, and a specialized evaluation suite (VFIG-BENCH) to achieve state-of-the-art performance comparable to GPT-5.2.

Original authors: Qijia He, Xunmei Liu, Hammaad Memon, Ziang Li, Zixian Ma, Jaemin Cho, Jason Ren, Daniel S Weld, Ranjay Krishna

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qijia He, Xunmei Liu, Hammaad Memon, Ziang Li, Zixian Ma, Jaemin Cho, Jason Ren, Daniel S Weld, Ranjay Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, hand-drawn blueprint of a house. It's perfect: you can move a wall, change a window, or resize a room with a single click because it's built from clean, mathematical lines. This is what SVG (Scalable Vector Graphics) is like for digital images.

Now, imagine someone takes a photo of that blueprint, prints it out, and then scans it back into a computer. Suddenly, it's just a flat picture (like a JPEG or PNG). The "walls" are no longer walls; they are just colored pixels. If you want to move a window now, you have to paint over the old one and draw a new one from scratch. It's messy, impossible to edit, and if you zoom in, it gets blurry.

The Problem:
In the world of science and engineering, researchers constantly lose these "blueprints." They have the final picture (the scan), but the original, editable code is gone. Manually redrawing these complex diagrams is like trying to rebuild a cathedral by hand, brick by brick, just because you lost the architect's plans. It takes forever and requires a specialist.

The Solution: VFig
The authors of this paper built a new AI called VFig (Vector Figure). Think of VFig as a super-intelligent "reverse engineer" that looks at a flat, messy photo of a diagram and instantly writes the original, clean, editable blueprint code for it.

Here is how they taught this AI to do it, using some creative analogies:

1. The Training Data: The "Cookbook"

To teach a chef to cook, you need recipes. To teach an AI to draw diagrams, you need a massive library of "Image + Code" pairs.

The Challenge: Existing libraries were like a cookbook full of simple recipes (like "how to draw a smiley face"). They didn't have complex recipes (like "how to draw a 10-step nuclear reactor").
The Fix: The team created VFig-Data, a massive new library with 66,000 complex diagrams. They didn't just copy-paste; they built a pipeline to filter out "bad ingredients" (like photos of clouds or math equations) and kept only the structural diagrams (flowcharts, architecture, networks). They also generated millions of synthetic diagrams to ensure the AI saw every possible shape and arrow combination.

2. The Training Method: "Learn to Walk, Then Run"

You wouldn't ask a baby to run a marathon on day one. You'd teach them to sit, then stand, then walk.

Step 1 (SFT - Supervised Fine-Tuning): The AI first practiced on simple diagrams (like basic shapes and arrows). This taught it the "alphabet" of SVG: how to draw a perfect circle, a straight line, or a text box.
Step 2 (The "Coarse-to-Fine" Curriculum): Once it mastered the alphabet, they gave it the "novels" (complex scientific papers). This helped it learn how to arrange those shapes into a coherent story (a flowchart or a system diagram).

3. The "Art Critic" (Reinforcement Learning)

Even after practice, the AI might draw a diagram that looks right but is structurally wrong (e.g., an arrow pointing to the wrong box).

The Old Way: Just check if the pixels match the original image. This is like judging a painting only by how close the colors are, ignoring if the perspective is wrong.
The VFig Way: They introduced a Reinforcement Learning stage. Imagine the AI draws a diagram, and then a strict Art Critic (another AI) looks at it. The Critic doesn't just say "Good job." It gives a report card:
- Did you include all the parts? (Presence)
- Are the boxes in the right spots? (Layout)
- Do the arrows connect the right things? (Connectivity)
- Is the text readable and the style correct? (Details)
If the AI gets a bad score, it tries again, learning from the Critic's feedback until it gets an "A." This is like a student taking a test, getting graded, studying the mistakes, and taking the test again until they master it.

4. The Results: Beating the Giants

The team tested VFig against other AI models and even massive, expensive, closed-source giants (like GPT-5.2 and Gemini).

The Outcome: VFig, which is open-source and free to use, performed on par with the most expensive, proprietary AI models. It could take a messy photo of a complex scientific diagram and turn it back into a clean, editable code file that looked almost identical to the original.

Why This Matters

Before VFig, if a scientist lost their diagram source file, that diagram was "dead"—it was just a picture. With VFig, that picture can be resurrected.

For Students: You can take a diagram from a textbook, turn it into code, and modify it for your own presentation.
For Engineers: You can update old technical manuals without needing the original CAD files.
For Everyone: It bridges the gap between a static image and a living, editable document.

In a nutshell: VFig is a time machine for digital diagrams. It takes a frozen, unchangeable snapshot and turns it back into a flexible, editable blueprint, saving researchers and designers from the nightmare of redrawing everything from scratch.

1. Problem Statement

The paper addresses the critical challenge of converting rasterized complex figures (e.g., PNG/JPEG images of scientific diagrams, flowcharts, and architecture layouts) back into editable, high-fidelity Scalable Vector Graphics (SVG) code.

Context: While SVG is the standard for technical illustration due to its resolution independence and semantic editability, original source files are frequently lost, leaving only flat raster images.
Challenges:
- Complexity: Real-world scientific figures involve nested layouts, heterogeneous primitives, precise alignments, and intricate connectivity, which are difficult to reconstruct.
- Data Scarcity: Existing datasets are small-scale, focusing on simple icons or decorative graphics rather than complex, structured diagrams.
- Generation Difficulty: Pure token prediction often fails to capture global layout consistency, topological edge cases (e.g., arrow connectivity), and fine-grained geometric details.
- Evaluation Gap: Traditional pixel-level metrics (e.g., SSIM, LPIPS) fail to capture structural integrity, while existing benchmarks lack complexity.

2. Methodology

The authors propose VFig, a family of Vision-Language Models (VLMs) trained specifically for figure-to-SVG conversion. The approach consists of three main pillars: Data, Training, and Evaluation.

A. Data Construction: VFig-Data

To overcome data scarcity, the authors curated VFig-Data, a large-scale dataset of 66,000 high-quality figure–SVG pairs.

Sources:
1. VFig-Data-Complex-Diagrams: Real-world scientific paper figures (flowcharts, architectures) collected from arXiv and converted to SVG using a two-stage VLM pipeline (Image $\to$ Structured Description $\to$ SVG Code).
2. VFig-Data-Shapes-and-Arrows: Programmatically generated diagrams using 19 layout templates with randomized shapes, connectors, and styles to ensure precise control over visual attributes.
Filtering: A rigorous filtering pipeline removes unsuitable images (natural photos, math equations, plots) and "path-heavy" SVGs (which lack semantic structure). The goal is to retain diagrams composed of semantic primitives (rects, circles, lines) rather than free-form Bézier paths.
Mixture: The final training mixture also includes 78K samples from existing academic datasets (SVG-Diagrams, Molmo2-Diagram) after similar filtering.

B. Training Strategy: Coarse-to-Fine Curriculum

The training process is designed to handle the compositional difficulty of complex figures through a two-stage approach:

Supervised Fine-Tuning (SFT):
- Curriculum: The model is first trained on structurally simpler diagrams (icons, basic shapes) to learn atomic primitives and basic layout. It is then fine-tuned on complex scientific figures to learn hierarchical compositions and global structure.
- Backbone: Uses Qwen3-VL (4B/8B) with LoRA (Low-Rank Adaptation).
Reinforcement Learning (RL) with Visual Feedback:
- Algorithm: Uses Group Relative Policy Optimization (GRPO).
- Reward Design: Instead of pixel-level loss, the model uses a rubric-based VLM judge (Gemini-3-Flash) to score generated SVGs against the ground truth. The reward is an average of four dimensions:
  - Presence: Are all elements present?
  - Layout: Is the spatial arrangement and alignment correct?
  - Connectivity: Do arrows/lines connect the correct endpoints?
  - Details: Are text, fonts, and styling accurate?
- This approach explicitly optimizes for structural fidelity and semantic correctness rather than just visual similarity.

C. Evaluation: VFig-Bench

The authors introduce VFig-Bench, a comprehensive benchmark specifically for complex figures, featuring:

Multi-Granularity Metrics:
- Pixel-level: SSIM, LPIPS, VisualSim (DINO/CLIP/SigLIP embeddings).
- Structural: Rule-based matching for shapes and arrows (in programmatically generated subsets).
- Semantic: VLM-Judge scores (Gemini and GPT-5.2) evaluating presence, layout, connectivity, and details.
Metrics for Code Quality: SVG Cleanliness (proportion of semantic primitives vs. paths) and Render Rate (percentage of successfully rendered code).

3. Key Contributions

VFig-Data: The first large-scale (66K), rigorously filtered dataset of complex scientific figures paired with structured, editable SVG code.
VFig Model: A VLM-based framework that achieves state-of-the-art (SOTA) performance among open-source models, rivaling proprietary systems like GPT-5.2.
Training Paradigm: A novel coarse-to-fine curriculum combined with RL using visual feedback (rubric-based rewards) that significantly improves structural consistency and layout accuracy over standard SFT.
VFig-Bench: A new evaluation suite with novel metrics that move beyond pixel similarity to measure structural integrity and semantic correctness.

4. Results

Experiments were conducted on three benchmarks: VFig-Bench (complex real-world figures), Molmo2-Diagram, and SVG-Diagrams.

Performance:
- VFig (SFT + RL) achieves the best overall performance among open-source models.
- It attains a VLM-Judge score of 0.829 on VFig-Bench, performing on par with GPT-5.2 (0.858) and outperforming other open-source baselines (e.g., OmniSVG, StarVector) by a large margin.
- It significantly outperforms classical raster-to-vector tools (like VTracer) in terms of SVG Cleanliness and Editability, as it generates semantic primitives rather than path-heavy traces.
Ablation Studies:
- Curriculum: Two-stage training (simple $\to$ complex) significantly improves render success rates and structural stability compared to one-stage training.
- RL Rewards: Using the full rubric-based reward (Presence, Layout, Connectivity, Details) yields better structural quality than pixel-level objectives. Adding pixel loss slightly improves SSIM but degrades judge-based structural scores.
- Backbone: Qwen3-VL-8B shows slight improvements over the 4B version, but the 4B model offers a better trade-off between performance and computational efficiency.

5. Significance

Bridging the Gap: VFig demonstrates that targeted data curation and structured training can narrow the performance gap between open-source models and massive proprietary systems for complex generative tasks.
Practical Utility: By producing clean, editable SVG code rather than path-heavy traces, VFig enables researchers and designers to easily modify, scale, and reuse scientific figures, accelerating revision workflows.
Methodological Insight: The paper establishes that structure-aware rewards (via VLM judges) are superior to pixel-level losses for optimizing complex diagram generation, providing a blueprint for future work in structured visual generation.
Open Science: The authors are releasing the models, data (VFig-Data), and evaluation tools (VFig-Bench) to foster further research in editable scientific figure generation.