💻 computer science

DiagramBank: A Large-scale Dataset of Diagram Design Exemplars with Paper Metadata for Retrieval-Augmented Generation

This paper introduces DiagramBank, a large-scale dataset of 89,422 curated schematic diagrams from top-tier scientific publications, designed to overcome the bottleneck of generating publication-grade teaser figures in autonomous AI scientist systems by enabling retrieval-augmented, exemplar-driven multimodal synthesis.

Original authors: Tingwen Zhang, Ling Yue, Zhen Xu, Shaowu Pan

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tingwen Zhang, Ling Yue, Zhen Xu, Shaowu Pan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant scientist who has just discovered a new way to cure a disease or build a better AI. You have written a fantastic paper explaining your work, but there's one huge problem: your paper looks like a wall of text.

In the world of science, the most important part of a paper isn't always the text; it's the "Teaser Figure." Think of this as the movie poster for your research. It's the single image on the first page that tells the whole story at a glance. It needs to be a map, a flowchart, or a blueprint that makes complex ideas look simple and exciting.

The problem? AI is great at writing text, but terrible at drawing these specific kinds of maps. Current AI tools can write a paragraph about a rocket ship, but if you ask it to draw the blueprint, it usually gives you a messy scribble or a generic cartoon.

This paper introduces DiagramBank, a solution to that problem. Here is the breakdown in simple terms:

1. The Problem: The "Blank Canvas" Panic

When an "AI Scientist" tries to write a paper, it can generate the code and the text, but when it gets to the diagram, it freezes. It doesn't know the "fashion rules" of science.

Analogy: Imagine asking a chef to cook a Michelin-star meal, but they've never seen a recipe or a picture of a plate. They might throw random ingredients together. It might taste okay, but it won't look like a professional dish. Scientists need their diagrams to look like they belong in a high-end gallery, not a kindergarten art project.

2. The Solution: A Massive "Pinterest" for Science

The authors built DiagramBank, which is essentially a giant, organized library of 89,422 beautiful, professional scientific diagrams taken from the world's top computer science conferences (like ICLR, NeurIPS, etc.).

How it works: They didn't just download pictures. They used a smart robot (an automated pipeline) to:
1. Read the papers.
2. Cut out the diagrams.
3. Filter out the junk (like simple bar charts or photos of people) and keep only the "blueprints" and "flowcharts."
4. Tag them with the story behind them (the title, the abstract, and the specific sentences in the paper that explain the drawing).

3. The Magic Trick: "Retrieval-Augmented Generation" (RAG)

This is the fancy term for the system's brain. Instead of asking the AI to "imagine" a diagram from scratch, the system acts like a super-smart librarian.

The Old Way: You ask the AI, "Draw me a diagram of a neural network." The AI guesses.
The DiagramBank Way: You tell the AI, "I'm writing a paper about neural networks."
1. The system searches DiagramBank and finds 5 real, perfect diagrams from past papers that are similar to your topic.
2. It shows these examples to the AI: "Look! This is how experts draw this. Copy the style, the colors, and the layout."
3. The AI then draws your new diagram, mimicking the style of those real examples.

Analogy: It's the difference between asking a student to draw a house without ever seeing one, versus showing them 10 photos of beautiful houses and saying, "Draw a new house that looks like these."

4. Why This Matters

For Scientists: It saves hours of frustration. You can get a "publication-ready" draft of your diagram instantly.
For AI: It teaches AI the "visual language" of science. It learns that arrows should point this way, that boxes should be grouped that way, and that scientific diagrams shouldn't look like comic books.
For the Future: It helps build "AI Scientists" that can write a complete paper, including the cover art, making the process of scientific discovery faster and more accessible.

The Catch (Limitations)

The authors are honest about the flaws:

The Filter isn't Perfect: Sometimes the robot might accidentally keep a chart that looks like a diagram, or miss a good one.
The Artist isn't Human Yet: Even with the examples, the AI might still draw a messy arrow or write text that is hard to read. It's a "draft," not a final masterpiece. You still need a human to check it.
Copyright: Since these diagrams come from real people's papers, you have to be careful about how you use them.

Summary

DiagramBank is a massive library of "how-to-draw" examples for scientists. It teaches AI how to stop drawing messy scribbles and start drawing professional, clear, and beautiful scientific maps, bridging the gap between writing a story and illustrating it.

1. Problem Statement

While autonomous "AI scientist" systems have made significant progress in generating code and scientific manuscripts, they face a critical bottleneck in producing publication-grade scientific diagrams (specifically "teaser figures" or method overview diagrams).

The Gap: Existing AI systems often omit these figures or generate inferior alternatives. Unlike standard data plots (which follow routine pipelines from data), scientific diagrams require conceptual synthesis, strategic layout planning, and the translation of complex logic into visual workflows.
Limitations of Current Models: Standard text-to-image models fail to generate dense, semantically linked diagrams with consistent arrow topology and readable text. They lack exposure to high-quality design exemplars that encode the visual conventions of scientific communication (e.g., module grouping, caption alignment).
Data Deficit: Existing datasets (e.g., SciCap, DocFigure, AI2D) are either optimized for caption generation, focused on educational diagrams, or lack the rich paper-level metadata required for domain-aware retrieval. There is no large-scale, public dataset specifically curated for exemplar-driven diagram design.

2. Methodology: Dataset Construction (DiagramBank)

The authors present DiagramBank, a large-scale dataset containing 89,422 schematic diagrams curated from top-tier AI/ML venues (ICLR, ICML, NeurIPS, TMLR) spanning 2017–2025. The construction follows a four-stage automated pipeline:

A. Data Acquisition

Source: PDFs and metadata extracted from OpenReview using the Python API.
Scope: 452,339 non-table figures extracted initially.
Metadata: Includes bibliographic details (title, abstract, authors, TL;DR), semantic content (keywords, subject areas), and platform-specific metrics (reviewer scores, decisions).

B. Content Extraction

Visuals: Uses PDFFigures 2.0 to parse PDFs and extract figures and captions. Tables are discarded.
Context: Uses PyMuPDF to extract text blocks that explicitly cite specific figure numbers (figure_context), linking the visual to the author's argumentative flow.

C. Schematic Classification (Filtering)

Challenge: Distinguishing explanatory diagrams (flowcharts, architectures) from data plots (bar charts) and natural images.
Solution: A CLIP-based filtering pipeline (OpenCLIP ViT-B-32).
- Images are classified into four categories: diagram, plot, image, other.
- Thresholding: Only figures classified as "diagram" with a confidence score > 0.85 are retained.
- Result: 89,422 high-fidelity schematic diagrams (19.8% of the total extracted corpus).

D. Data Aggregation

Structure: Data is denormalized using DuckDB. Each figure record is paired with its parent paper's metadata (title, abstract) and figure-level context (caption, in-text references).
Output: A ready-to-index format where every record is a self-contained unit, facilitating retrieval without complex runtime joins.

3. System Overview: DiagramBank-RAG

To demonstrate the utility of the dataset, the authors developed DiagramBank-RAG, a retrieval-augmented generation framework for scientific diagram authoring.

Goal: Given a paper title/abstract and a target figure description, retrieve relevant design exemplars to guide a generative model.
Hierarchical Three-Stage Retrieval Strategy:
1. Stage 1 (Title Filtering): Coarse-grained filtering using the paper title to constrain the search space to the target discipline ( $S_1$ ).
2. Stage 2 (Abstract Refinement): Deep fetch within $S_1$ using the paper abstract to match problem settings and methodology ( $S_2$ ).
3. Stage 3 (Caption Matching): Fine-grained matching using the figure caption to retrieve specific visual exemplars aligned with the user's intended diagram description.
Generation: Retrieved exemplars serve as visual priors (style, layout, iconography) for a downstream generative model (e.g., Nano Banana 3 Pro), enabling exemplar-conditioned synthesis.

4. Key Results and Analysis

Dataset Scale:
- Total Diagrams: 89,422 (High-confidence subset: 59,765).
- Venues: ICLR, ICML, NeurIPS, TMLR (2017–2025).
- Distribution: Diagrams constitute ~~19.8% of all extracted figures, with a stable ratio across venues (~~18–21%).
Visual Density:
- Papers contain an average of 4.79 to 9.22 figures.
- Visual breadth is high (79.8%–97.6% of papers have at least one extractable figure).
Case Study (Code2MCP Framework):
- Baseline (Prompt-only): Generated a generic "DMemphis" style diagram with high-contrast colors and linear layout. It captured topology but lacked academic aesthetic fidelity.
- RAG-Augmented: Successfully retrieved stylistic exemplars (pastel palettes, rounded containers, cyclical workflows). The output adopted the professional aesthetic, structural coherence, and specific iconography of the retrieved references, demonstrating that retrieval improves layout plausibility and adherence to scientific conventions.

5. Key Contributions

Large-Scale Dataset: Release of DiagramBank, the first large-scale dataset of 89k+ schematic diagrams paired with rich paper metadata and figure-local context, specifically designed for multimodal retrieval.
Contextual Enrichment: Each diagram is linked to multi-level text signals (Title/Abstract for intent; Caption/In-text references for semantics), enabling hierarchical retrieval that reduces domain drift.
Baseline Systems: Provision of baseline code for indexing, retrieval, and a RAG-style pipeline demonstrating how retrieved exemplars can guide the synthesis of teaser-style diagrams.
Infrastructure: The dataset is released in a ready-to-index format (HuggingFace) with documentation for immediate use in scientific authoring systems.

6. Significance and Future Work

Significance: DiagramBank bridges the gap between text-centric writing pipelines and figure-centric scientific communication. It provides the necessary "visual vocabulary" for AI agents to generate publication-ready diagrams, moving beyond generic image generation to domain-specific design.
Limitations:
- Reliance on automatic heuristics may introduce label noise.
- Current image generators still struggle with dense arrow topology and legible text, often requiring post-editing.
- Potential sampling bias due to reliance on OpenReview (open-access venues).
Future Directions: Developing stronger discriminators for exemplar selection, utilizing explicit intermediate representations (e.g., layout programs) for controllable synthesis, and expanding coverage to diverse domains.

Conclusion: DiagramBank establishes a new infrastructure for the "AI Scientist" paradigm, enabling the retrieval of high-quality design patterns to automate the creation of the most critical visual component of a scientific paper: the teaser figure.