💻 computer science

LiveFigure: Generating Editable Scientific Illustration with VLM Agents

LiveFigure is an agentic framework driven by Vision-Language Models that mimics human researchers' multi-step workflow to generate fully editable, vectorized scientific illustrations via PowerPoint scripts, significantly outperforming existing baselines in publication readiness and human preference.

Original authors: Chenyang Shao, Jiahe Liu, Fengli Xu, Yong Li

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenyang Shao, Jiahe Liu, Fengli Xu, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist who has just discovered a brilliant new idea. You need to draw a diagram to explain it to the world. In the past, you'd have to spend hours manually dragging boxes, drawing arrows, and tweaking fonts in software like PowerPoint or Illustrator. It's tedious, like building a house brick-by-brick with tweezers.

Recently, AI image generators (like the ones that make pretty pictures from text) promised to do this for you. But they had a major flaw: they painted the picture on a canvas and then sealed it with varnish. Once the picture was made, you couldn't move a single box or fix a typo without painting the whole thing over again. If you wanted to move the "Weighted Sum" box down, the AI would just try to redraw the entire image, often getting the rest of the picture wrong.

LiveFigure is a new system that changes the game. Instead of painting a static picture, it acts like a super-smart, hyper-organized architect who builds a fully editable model of your diagram.

Here is how it works, broken down into three simple steps:

1. The "Blueprint" Phase (Visual Planning)

Before building anything, a human architect looks at successful buildings to see what works. LiveFigure does the same. It scans thousands of high-quality scientific diagrams from top journals to learn the "rules of the road."

The Analogy: Imagine you are building a treehouse. Instead of guessing where the branches go, you look at photos of the best treehouses ever built. You learn that the ladder usually goes on the left and the window faces the sun. LiveFigure uses this "visual memory" to sketch a mental blueprint of your diagram before it writes a single line of code.

2. The "Construction" Phase (Procedural Generation)

This is where LiveFigure differs from other AI. Instead of painting pixels, it writes code (specifically, Python scripts that talk to Microsoft PowerPoint).

The Analogy: Think of other AI as a painter who mixes colors on a palette. LiveFigure is a robotic carpenter with a toolbox of pre-made, perfect parts. It doesn't try to "guess" how to draw a box; it uses a pre-tested tool called add_box(). It doesn't guess how to draw an arrow; it uses a tool called add_connector().
The "Experience" Trick: Sometimes, even robots make mistakes (like trying to use a hammer as a screwdriver). LiveFigure keeps a "diary of mistakes." If it tries to do something that crashed the program before, it remembers: "Never do that again!" This helps it write code that actually works the first time.

3. The "Quality Check" Phase (Targeted Refinement)

Once the robot builds the diagram, a "Visual Inspector" (another AI) looks at the result.

The Analogy: Imagine the carpenter builds a shelf, but the inspector notices a screw is sticking out or a shelf is slightly crooked. Instead of tearing the whole shelf down and starting over, the inspector points to the specific screw and says, "Turn this one clockwise."
LiveFigure makes these tiny, precise adjustments to the code. It fixes the crooked arrow or the overlapping text box without touching the rest of the diagram.

The Result: A "Lego" Diagram

The final output is not a flat image (like a JPEG). It is a PowerPoint file where every single box, arrow, and word is a separate, movable object.

Why this matters: If you want to change the color of one box, you just click it and change the color. If you want to move a whole section, you drag it, and the arrows automatically stretch and bend to follow it. It's like building with Lego bricks instead of pouring concrete.

What the Paper Found

The researchers tested LiveFigure against the best image-generating AI models (like Google's "Nano Banana" and OpenAI's "GPT-Image") and traditional coding tools.

The "Fix-It" Test: They asked humans to take the AI-generated diagrams and make them ready for a scientific paper.
- Other AI: The human had to make about 30+ edits to fix the mess, and even then, only 24% of the diagrams were good enough to use.
- LiveFigure: The human only needed to make about 17 edits (mostly small tweaks), and 80% of the diagrams were ready to publish immediately.
Human Preference: When people were shown two diagrams side-by-side (one from LiveFigure, one from the competition), they picked LiveFigure 60% of the time because it looked more professional and made more sense logically.

In Short

LiveFigure doesn't just "draw" a picture; it builds a diagram using a smart, step-by-step process that mimics how human experts think. It creates a diagram that you can actually edit, fix, and perfect, turning the difficult task of scientific illustration into something that feels like playing with digital Legos.

Technical Summary: LiveFigure

Problem Statement
Scientific illustrations are critical for communicating complex methodologies and conceptual frameworks, yet their creation remains a labor-intensive process requiring deep domain knowledge and proficiency with professional design tools. While recent generative image models have advanced prompt-based editing, they fundamentally generate static, pixel-based raster images. This paradigm presents two primary limitations for scientific publishing:

Lack of Editability: Raster outputs treat figures as indivisible visual blocks. Correcting minor structural inaccuracies, spelling errors, or layout misalignments requires full regeneration via iterative natural language prompts, which fails to support the fine-grained, verifiable editing process essential in scientific writing.
Format Incompatibility: Top-tier academic venues (e.g., Nature, Science) mandate vector formats for submission to ensure resolution-independent scaling and native editability. Existing generative models cannot produce vector graphics, and current workarounds (e.g., layer decomposition or code-based generation) either remain pixel-based or produce visually simplistic, non-editable outputs.

Methodology: Cognition-Inspired Procedural Construction
The authors propose LiveFigure, an agentic framework driven by Vision-Language Models (VLMs) that abandons end-to-end pixel generation in favor of a "Cognition-Inspired Procedural Construction" paradigm. This approach mimics the staged cognitive workflow of human experts: retrieving references, conceptualizing blueprints, assembling assets, and refining based on visual feedback. The framework operates through three sequential stages:

Visual Planning via Prior Induction ( $\Psi_{plan}$ ):
- The system constructs a knowledge base of high-quality methodological figures from top-tier conferences (ICLR, NeurIPS, ICML).
- A retrieve agent fetches semantically relevant reference figures based on the input methodological text.
- A VLM analyzes these references to distill design priors and layout structures, generating a tailored structural plan and a visual blueprint.
- An asset generation module synthesizes style-consistent icons for complex entities using a grid-based batch generation strategy.
Procedural Figure Generation via Skills and Experience ( $\Psi_{assemble}$ ):
- LiveFigure selects Microsoft PowerPoint as the output carrier due to its ubiquity in research, open XML structure, and programmatic accessibility via python-pptx.
- Instead of prompting raw code, the system utilizes a Standardized Skill Library ( $S$ ) containing pre-validated, high-level semantic functions (e.g., connector routing, nested text-shape composition).
- An Experience Injection mechanism incorporates accumulated debugging constraints (negative constraints) to prevent known API hallucinations and parameter errors.
- A self-correcting execution loop handles runtime errors by feeding stack traces back to the model for iterative code refinement until a valid script is produced.
Targeted Refinement via Visual Diagnostics ( $\Psi_{refine}$ ):
- To bridge the gap between code logic and visual perception, a "visual critic" VLM inspects the rendered figure.
- It identifies subtle defects invisible to code logic (e.g., element occlusion, misalignment, inconsistent styling) and generates a structured list of actionable issues.
- The system applies incremental code-level updates to address these specific issues, iterating until the figure meets publication standards.

Key Contributions

Novel Paradigm: Introduction of "Cognition-Inspired Procedural Construction," which shifts scientific figure generation from pixel manipulation to a staged, code-driven workflow that ensures inherent editability.
LiveFigure Framework: A practical multi-agent system that produces fully editable, vectorized PowerPoint source files, overcoming the limitations of raster-based models and the rigidity of traditional code-based tools.
Empirical Validation: Extensive experiments demonstrating that the framework generates figures requiring minimal manual intervention to reach publication readiness.

Experimental Results
The framework was evaluated on a dataset of 300 scientific schematic figures derived from accepted papers at ICLR 2024, NeurIPS 2024, and ICML 2024.

Editability: LiveFigure achieved an 80% publication-readiness rate with a median of only 17 manual edits (Edit Distance). This significantly outperformed the strongest baseline, NanoBanana (Gemini-3-Pro-Image), which achieved only a 24% rate.
Human Preference: In double-blind pairwise comparisons, LiveFigure secured a 60% win rate against NanoBanana and a 69% win rate against GPT-Image-1.5, with all confidence intervals strictly exceeding the 50% threshold.
Quality Metrics: LiveFigure demonstrated superior performance in Content Fidelity (Accuracy: 8.29, Completeness: 8.16 under V2 input) compared to raster generators, while maintaining competitive Visual Design scores. It outperformed code-based tools (e.g., Matplotlib, Graphviz) in aesthetic quality and layout flexibility.
Ablation Studies: Removing the Skill Library caused a catastrophic drop in code executability (from 83.5% to 40.0%), while removing the Experience module increased the median edit distance by nearly 4x, confirming the necessity of both components.

Significance and Claims
The paper claims that LiveFigure fundamentally addresses the "editability gap" in scientific illustration by decoupling generation from pixel manipulation. By leveraging PowerPoint as an intermediate, editable vector format, the system democratizes high-quality figure creation, lowering the technical barrier for researchers across disciplines. The authors assert that their approach enables seamless human-AI collaboration, where the AI handles labor-intensive structural and spatial layout tasks, leaving the final aesthetic refinement to the user. The work positions itself as a specialized agent that transforms abstract research designs into publication-standard, editable figures, bridging the gap between automated research assistance and the rigorous visual standards of academic publishing.