🤖 AI

SciFig: Towards Automating Editable Figure Generation for Scientific Papers

The paper introduces SciFig, an end-to-end multi-agent framework that overcomes the trade-off between visual quality and editability by automatically generating high-quality, fully editable methodology figures from scientific text, alongside a new benchmark (SciFig-Bench) and evaluation protocol (SciFig-Eval) to validate its superior performance.

Original authors: Siyuan Huang, Yifan Zhou, Yutong Gao, Zi Yin, Juyang Bai, Xinxin Liu, Rama Chellappa, Chun Pong Lau, Cheng Peng, Sayan Nag, Shraman Pramanick

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyuan Huang, Yifan Zhou, Yutong Gao, Zi Yin, Juyang Bai, Xinxin Liu, Rama Chellappa, Chun Pong Lau, Cheng Peng, Sayan Nag, Shraman Pramanick

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are writing a recipe for a complex new dish. You've written down all the steps in words, but you know that to really help a chef understand how the ingredients flow from the bowl to the stove, you need a picture.

In the world of scientific research, these "pictures" are called methodology figures. They are the diagrams that show how a new computer program or scientific method actually works.

The Problem:
Right now, making these diagrams is like trying to draw a masterpiece with a crayon while wearing oven mitts.

The "Crayon" Problem: If you use standard drawing tools (like PowerPoint), you can edit the picture easily, but it often looks stiff, flat, and a bit boring—like a stick-figure sketch.
The "Oven Mitt" Problem: If you use fancy AI image generators to make it look beautiful and realistic, you get a high-quality picture, but it's like a photograph. If you want to change just one ingredient or move one arrow, you can't just click and drag. You have to throw the whole picture away and start over.

The Solution: SciFig
The authors of this paper built a new system called SciFig. Think of SciFig as a team of four specialized chefs working together to build a custom, editable, and beautiful diagram from your written recipe.

Here is how the team works:

The Planner (The Architect): First, this agent reads your text and figures out the "blueprint." It decides, "Okay, this part goes on the left, that part goes on the right, and these two need to be connected by a big arrow." It doesn't draw anything yet; it just makes the plan.
The Layout Agent (The Builder): This agent takes the blueprint and builds the frame. It creates a structured skeleton (using a digital format called XML) that holds the pieces in place. Because it's a skeleton and not a finished painting, you can still move the walls around easily.
The Component Agent (The Decorator): Now, this agent fills in the details. It adds the "fancy" parts—like realistic textures, colors, and icons—making the diagram look professional and rich. But, it attaches these decorations to the skeleton, so they stay editable.
The Feedback Agent (The Critic): Finally, this agent looks at the result. It can listen to a human saying, "Move that box up," or it can use its own "eyes" (an AI vision model) to say, "Hey, these two arrows are crossing awkwardly; let's fix that." It then tweaks the diagram until it's perfect.

The Result:
The final product is a diagram that looks as good as a human expert drew it, but it remains fully editable. You can click on a box, change a label, or move an arrow without ruining the whole image. The system does this in about 10 minutes.

How They Tested It (SciFig-Bench & SciFig-Eval)
To prove their system works, the team didn't just guess. They:

Built a Library (SciFig-Bench): They gathered 435 real, high-quality diagrams from top computer science papers. These served as the "gold standard" answers.
Created a Grading Rubric (SciFig-Eval): Instead of just asking an AI, "Is this pretty?" they created a strict 4-point checklist:
1. Completeness: Did it include all the important parts?
2. Fidelity: Does it look like the original author's drawing?
3. Quality: Is the text clear and the layout logical?
4. Design: Are the colors and spacing pleasing to the eye?

The Verdict:
When they compared SciFig to other methods (including single AI image generators and other multi-step systems), SciFig won every category.

It produced diagrams that were more accurate and complete.
It looked better and was easier to read.
Most importantly, in a blind test with 60 human experts, SciFig was chosen as the best 81% of the time, beating even the original diagrams drawn by the paper authors themselves.

In Short:
SciFig is a tool that automates the hard work of drawing scientific diagrams. It combines the editability of a digital drawing tool with the visual beauty of an AI image generator, ensuring that scientists can communicate their ideas clearly without spending hours wrestling with drawing software.

Technical Summary: SciFig

Problem Statement

High-quality methodology figures are essential for scientific communication, serving to distill complex methods, components, and information flows into clear, revisable diagrams. However, creating these figures remains a manual, time-consuming process. Existing automated approaches face a fundamental trade-off:

Structured/Editable Approaches: Methods based on TikZ or SVG produce editable outputs but often lack the visual richness, texture, and stylistic variation of human-designed illustrations.
Image-Generation Approaches: Models leveraging VLMs and image generators produce polished, publication-ready raster images but output non-editable files where even minor revisions require full regeneration.

Furthermore, evaluating generated scientific figures is challenging. Current evaluation protocols often reduce multidimensional quality (correctness, completeness, readability, design) to a single numerical score from a single VLM, which introduces bias and sensitivity to prompt phrasing.

Methodology: SciFig Framework

The authors introduce SciFig, an end-to-end multi-agent framework designed to generate visually rich, fully editable methodology figures from scientific text. The system bridges the gap between raster expressiveness and symbolic editability by producing XML figures that can be edited in standard diagramming tools (e.g., draw.io, PowerPoint).

System Architecture

SciFig decomposes the generation process into four stages, each handled by a dedicated agent:

Planning Agent: Analyzes the methodology text ( $T$ ) and figure description ( $D$ ) to produce a textual plan ( $P$ ). This plan specifies layout direction, aspect ratio, pipeline stages, functional groupings, and internal connections, separating what to depict from where elements should appear.
Layout Agent: Converts the plan $P$ into an initial editable XML layout ( $L_0$ ). This layout encodes positions, sizes, and connections using a three-level hierarchy (modules, subgroups, leaf components) and uses thick/thin arrows to denote inter-module and intra-module flows.
Component Agent: Populates the layout with visual elements ( $V$ $V$ ). It employs a hybrid rendering strategy:
- PNG Mode: For visually complex components (e.g., heatmaps), an image-generation model creates a rich element embedded as a base64 image within the XML.
- Textbox Mode: For named operations and variables, styled labeled rectangles are generated natively in the vector format.
Feedback Agent: Refines the figure through iterative loops ( $K$ $K$ rounds). It supports two modes:
- Human-in-the-loop: Researchers provide direct edits or natural language instructions.
- VLM-in-the-loop: A VLM critiques the rendered figure for layout, alignment, and clarity, suggesting revisions.

The final output is a structured XML file representing $F = (V, L, E)$ , allowing individual components to be repositioned or restyled without regenerating the entire figure.

Key Contributions

1. SciFig Framework

A multi-agent system that generates methodology figures balancing visual richness and full editability. It supports both human and VLM feedback loops for refinement.

2. SciFig-Bench

A human-verified benchmark consisting of 435 methodology figures and their corresponding source text/captions. The dataset spans 37 AI/ML-centered domains across 15 top-tier venues (e.g., NeurIPS, ICML, CVPR). An additional set of 1,784 figures was used to develop evaluation rubrics.

3. SciFig-Eval

A four-axis evaluation protocol designed to overcome the limitations of single-score VLM judging:

Completeness and Correctness: Uses cross-model Visual Question Answering (VQA) with 50 yes/no questions to assess information coverage.
Reference-Based Fidelity: Measures visual similarity (DINOv2), figure-text relevance (SigLIP2), and OCR text accuracy (F1 score) against author-drawn references.
Rubric-Based Content Quality: Employs 20 binary questions derived from real figures to evaluate technical accuracy, clarity, consistency, and readability.
Perceptual Design Quality: Uses 8 deterministic heuristics (e.g., WCAG contrast, data-ink ratio, whitespace uniformity) to assess low-level design properties without reference bias.

Experimental Results

The authors evaluated SciFig against seven baselines, including single-agent image generators (GPT-5.4-Image, Qwen-Image, Nano Banana Pro) and agentic pipelines (Paper2Poster, AutoFigure, etc.).

Performance: SciFig achieved the highest scores across all four SciFig-Eval axes. Notably, it outperformed the best baseline (PaperBanana) by 0.4 points in Completeness & Correctness and 0.4 points in Rubric-Based Content Quality.
Editability: Unlike raster generators, SciFig produces XML figures where labels and symbols remain editable and legible, avoiding the blurring or corruption common in image-generation outputs.
Human Evaluation: In a blind pairwise study with 60 participants (10,000 comparisons), SciFig ranked first with an Elo rating of 1327 and an 81.4% win rate, outperforming even the original author-drawn figures (GT Figures).
Efficiency: The system generates editable figures in approximately 10 minutes on average.
Ablation: The study confirmed that the Layout Agent and Feedback Agent significantly improve structural consistency and overall quality. Increasing CoT refinement rounds ( $K$ ) from 1 to 4 yielded diminishing returns after the third round.

Significance and Claims

The paper claims that SciFig represents a significant step forward in automating scientific visual communication by resolving the trade-off between editability and visual quality. By decomposing figure generation into planning, layout, rendering, and refinement, the framework produces figures that are not only faithful to the source text but also adaptable to the iterative nature of scientific publishing.

The authors emphasize that SciFig is designed primarily for methodology figures within AI/ML domains. While qualitative examples suggest generalization to teaser diagrams and statistical plots, the authors explicitly state that they do not systematically benchmark these other figure types. Similarly, the benchmark is limited to AI/ML venues, and the authors acknowledge that visual conventions in other scientific fields (e.g., chemistry, biology) may require domain-specific adaptations.

The work positions SciFig as an assistive drafting tool intended to support researchers, particularly those without specialized design training, while maintaining author control over the final output. The authors caution that generated figures should not be treated as a substitute for authorial judgment or technical correctness.