💬 NLP

Text2Arch: A Dataset for Generating Scientific Architecture Diagrams from Natural Language Descriptions

This paper introduces Text2Arch, a comprehensive dataset of scientific architecture diagrams paired with textual descriptions and DOT code, which enables the training of language models to generate high-fidelity diagrams from natural language, achieving performance comparable to GPT-4o while significantly outperforming existing baselines.

Original authors: Shivank Garg, Sankalp Mittal, Manish Gupta

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shivank Garg, Sankalp Mittal, Manish Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an architect trying to explain a massive, complex skyscraper to a builder. You could write a 50-page manual describing every beam, window, and electrical wire. But the builder would likely get lost, confused, or miss a crucial detail. It's much better to just hand them a clear, blueprinted drawing.

The Problem:
In the world of science and technology, researchers often describe complex systems (like how an AI brain works or how a software company is organized) using only words. This is like trying to build that skyscraper from the 50-page manual. It's slow, prone to errors, and hard to visualize.

The Solution: TEXT2ARCH
This paper introduces a new tool called TEXT2ARCH. Think of it as a "Magic Translator" that takes a paragraph of text describing a system and instantly turns it into a clean, professional blueprint (a diagram).

Here is how the paper breaks down this idea, using some everyday analogies:

1. The Missing Recipe Book (The Dataset)

Before you can teach a robot to cook, you need a massive library of recipes and the finished dishes. The authors realized that while we have lots of pictures of diagrams and lots of text, nobody had ever put them together in a clean, organized way.

The Analogy: Imagine trying to teach a chef to bake a cake, but you only have a pile of flour and a pile of pictures of cakes, with no instructions linking the two.
The Fix: The team created TEXT2ARCH, a massive "cookbook" containing over 75,000 examples. Each example has three parts:
1. The Description (The recipe text).
2. The Image (The finished cake).
3. The Code (The specific instructions the computer needs to draw the cake).

2. The Three-Step Assembly Line (How they made the data)

They didn't just copy-paste these 75,000 examples; they built a smart factory to create them:

Step 1: The Filter (The Bouncer): They took a huge pile of scientific papers and used a smart AI "bouncer" to let in only the diagrams that looked like blueprints (architecture) and kick out the ones that were just charts or photos.
Step 2: The Translator (The Scribe): They used a super-smart AI (GPT-4o) and a "robot eye" (OCR) to look at the diagrams and write down the code (called DOT code) that tells a computer how to draw them. It's like having a scribe who looks at a map and writes down the exact GPS coordinates for every turn.
Step 3: The Editor (The Polisher): They used the AI again to clean up the text descriptions, making sure they were clear, concise, and perfectly matched the diagram.

3. The Race (The Experiments)

Now that they had the "cookbook," they wanted to see if they could teach a computer to draw these diagrams from scratch. They entered a race against other methods:

The Old Guard (DiagramAgent): A previous system that tried to do this but was a bit clumsy.
The Big Shot (GPT-4o): A very powerful, expensive AI that was asked to do the job without any special training (Zero-Shot).
The Underdogs (Small Models): Smaller, cheaper AI models (like DeepSeek, Llama, and Qwen) that were fine-tuned (trained specifically) on the TEXT2ARCH cookbook.

The Result:
The fine-tuned small models won the race!

The Analogy: It's like taking a local chef (the small model), giving them a specific, high-quality cookbook (TEXT2ARCH), and having them beat a famous celebrity chef (GPT-4o) who had to guess the recipe on the fly.
The small models learned the "language" of diagrams so well that they could generate code that was more accurate and structured than the big, expensive models.

4. Why This Matters (The Impact)

Why do we care if a computer can draw a diagram?

For Students: Imagine reading a textbook and instantly seeing a 3D model of the concept you just read.
For Engineers: If a software engineer changes a few lines of code in their description, the diagram updates automatically. No more manual redrawing!
For Researchers: It turns a wall of text into a visual map, making it easier to spot mistakes or understand complex ideas quickly.

The Bottom Line

The paper says: "We built a massive, high-quality library of text-and-diagram pairs. We used it to train smaller, smarter computers to turn words into blueprints. These computers are now better at drawing technical diagrams than the biggest, most expensive AI models we have today."

It's a giant leap toward making the invisible world of complex systems visible, understandable, and easy to build.

1. Problem Definition

The paper addresses the challenge of automatically generating scientific architecture diagrams from natural language text descriptions.

The Gap: While Large Language Models (LLMs) and diffusion models have advanced text-to-image generation, they struggle with structured diagrams (e.g., system architectures, neural network flows). Diffusion models often fail to maintain strict semantic alignment, logical coherence, and precise textual labels, producing unreadable or structurally incorrect diagrams.
The Specific Task: The authors define the TEXT2ARCH task: translating a textual description of a system (e.g., a microservices pipeline or a neural network) into a DOT code representation (a graph description language), which is then compiled into a visual architecture diagram.
The Bottleneck: The field lacked a large-scale, high-quality, open-access dataset containing aligned triplets of Text $\to$ DOT Code $\to$ Image specifically for scientific architecture diagrams. Existing datasets were either too heterogeneous (mixing plots, tables, and diagrams) or lacked clean code-grounding.

2. Methodology

A. Dataset Curation: TEXT2ARCH

The core contribution is the creation of the TEXT2ARCH dataset, comprising 75,127 samples. The curation pipeline involved three main stages:

Architecture vs. Non-Architecture Classification:
- The authors trained a binary classifier (using CLIP, ViT, BEiT, ResNet) to filter general scientific figures into architecture diagrams.
- Training data was synthesized from ACL-Fig, SciFig, and manually labeled subsets of Paper2Fig.
- The best model (CLIP) achieved 83.45% test accuracy with high recall (0.92), successfully filtering the Paper2Fig dataset to isolate ~80k architecture images.
DOT Code Generation (Ground Truth):
- Since the source images lacked code, a multi-step pipeline generated the ground-truth DOT code:
  - DOT1: Direct GPT-4o generation from the image (often noisy/incomplete).
  - DOT2: Object detection (Faster-RCNN) + OCR (Florence-2) to extract nodes, labels, and arrow directions.
  - DOT3 (Final Ground Truth): GPT-4o refinement of DOT2 using the image as a reference to correct structural errors and ensure semantic coherence.
Description Refinement:
- Desc1: Original figure captions/paragraphs from papers.
- Desc2: Retrieval of relevant paragraphs from the paper using TF-IDF similarity.
- Desc3 (Final Input): GPT-4o refined description combining Desc1 and Desc2 to create a clean, concise, and semantically rich text prompt.

The dataset is stratified by complexity (number of nodes): Easy (54.3%), Medium (32.4%), and Hard (13.3%).

B. Model Approaches

The authors benchmarked several approaches for the Text-to-DOT task:

Baselines:
- DiagramAgent: A multi-agent framework (converted from TikZ to DOT for comparison).
- Zero-shot GPT-4o: Direct inference without fine-tuning.
Few-Shot In-Context Learning (ICL):
- Tested on instruction-tuned small language models (SLMs): Llama-3-8B, Qwen2-7B, and DeepSeek-LLM-7B.
Supervised Fine-Tuning (SFT):
- The same SLMs were fine-tuned on the TEXT2ARCH training set (60,519 samples) to specialize in generating structured DOT code from technical descriptions.

C. Evaluation Metrics

The paper proposes a dual-metric evaluation framework:

Standard NLG Metrics: ROUGE-L, CodeBLEU, Levenshtein Edit Distance, and chrF (character n-gram F-score) to measure code similarity.
Graph-Based Metrics:
- Node/Edge Precision, Recall, F1: Calculated using optimal matching (Hungarian algorithm) based on label similarity.
- PR-AUC: Area under the Precision-Recall curve for node/edge matching across varying similarity thresholds.
- Jaccard Similarity: Measuring edge overlap between predicted and ground-truth graphs.

3. Key Contributions

Novel Task Definition: Formalized the generation of scientific architecture diagrams via intermediate DOT code, bridging the gap between unstructured text and structured visual logic.
TEXT2ARCH Dataset: Released a large-scale (75k+), high-quality dataset with clean text-code-image triplets, categorized by complexity.
Model Performance: Demonstrated that fine-tuned small language models (specifically DeepSeek-7B) significantly outperform both few-shot ICL models and existing baselines like DiagramAgent.
Evaluation Framework: Introduced a rigorous graph-level evaluation suite (Node/Edge F1, PR-AUC) specifically designed to assess structural fidelity, not just text similarity.

4. Results

Superiority of Fine-Tuning: Fine-tuned models drastically outperformed zero-shot and few-shot approaches.
- On the Manual Annotation Set, the fine-tuned DeepSeek-7B achieved a ROUGE-L of 55.2 and CodeBLEU of 49.3, compared to 28.2 and 16.3 for GPT-4o, and 49.1 and 40.9 for DiagramAgent.
- In Graph Metrics, DeepSeek-7B achieved the highest Node F1 (69.4) and Edge F1 (49.1) on the manual set, indicating superior structural accuracy.
Comparison with Baselines:
- DiagramAgent: Struggled with structural fidelity, often producing incorrect edges or generic nodes.
- Few-Shot ICL: Performed inconsistently; while Llama-3 and Qwen2 showed some improvement, DeepSeek's few-shot performance was notably worse than its fine-tuned counterpart.
Human Evaluation: In a preference study, human evaluators preferred the outputs of the fine-tuned DeepSeek-7B model over DiagramAgent in 71.6% of cases.
Robustness: The model maintained high performance even with variations in description length (shortened or lengthened), though extreme shifts in length caused some degradation.

5. Significance and Impact

Productivity Boost: The system enables rapid conversion of technical documentation, research papers, and patent drafts into visual architecture diagrams, saving significant manual effort.
Educational & Enterprise Utility: Facilitates the creation of educational materials and helps in visualizing complex enterprise systems for better stakeholder understanding.
Open Science: By releasing the dataset, code, and models, the authors lower the barrier for research in structured diagram generation, a field previously hindered by data scarcity.
Paradigm Shift: The work suggests that for structured, logic-heavy generation tasks, fine-tuned small models can outperform massive proprietary models (like GPT-4o) and complex multi-agent frameworks, offering a more efficient and controllable solution.

Limitations Noted: The current work is limited to English; it relies on GPT-4o for ground-truth generation (potential error propagation); and it struggles with extremely dense graphs or ambiguous inputs. Future work may explore multilingual support and reinforcement learning for alignment.