🤖 machine learning

Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks

This paper introduces GraphicDesignBench (GDB), the first comprehensive benchmark suite evaluating AI models on professional graphic design tasks across five axes—layout, typography, infographics, templates, and animation—revealing that while high-level semantic understanding is achievable, current models still significantly struggle with the precision, structural validity, and compositional reasoning required for real-world design work.

Original authors: Adrienne Deganutti, Elad Hirsch, Haonan Zhu, Jaejung Seol, Purvanshi Mehta

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adrienne Deganutti, Elad Hirsch, Haonan Zhu, Jaejung Seol, Purvanshi Mehta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of incredibly smart, super-fast robots. You've taught them to write poetry, solve math problems, and even draw pictures of cats and dogs that look almost real. You might think, "Great! Let's ask them to be graphic designers for our company."

But before you hand them the keys to your design studio, you need to know: Can they actually do the job?

That's exactly what this paper, GraphicDesignBench (GDB), is about. It's like a giant, rigorous "final exam" for AI, specifically designed to test if robots can handle the messy, detailed, and precise world of professional graphic design.

Here is the breakdown of what they found, using some everyday analogies.

The Problem: The "Cat vs. Brochure" Gap

Think of current AI image generators (like the ones that make cool art from text) as abstract painters. They are great at making a picture that looks nice. If you ask for "a sunset," they give you a beautiful sunset.

But a professional graphic designer is more like an architect or a carpenter. They don't just make things look pretty; they have to follow strict blueprints.

The text must be spelled correctly.
The logo must be in the exact right spot.
The font must match the brand guidelines perfectly.
The layers must be organized so a human can edit them later.

The paper argues that while AI is getting good at being an "abstract painter," it is terrible at being a "carpenter."

The Exam: 50 Tasks, 5 Categories

The researchers created a test with 50 different challenges based on real-world design templates. They tested the AI in five main areas:

Layout (The Floor Plan): Can the AI look at a poster and tell you how many items are on it? Can it tell you which item is "on top" of another?
- The Result: The AI is like a child looking at a messy room. It can guess there are "some things" there, but it can't count them accurately, and it often thinks the floor is the ceiling. It struggles to understand the 3D stacking of 2D images.
Typography (The Fonts): Can the AI look at a headline and tell you exactly what font is used, how big the letters are, and what color they are?
- The Result: The AI is like someone who knows what a "red" shirt looks like but can't tell the difference between a "crimson" and a "scarlet" shirt. It often guesses the wrong font family entirely or gets the letter spacing wrong. It's like trying to read a menu written in a language you don't speak.
Infographics & Vectors (The Blueprints): Can the AI take a description and write the actual computer code (SVG) to draw a perfect icon?
- The Result: This is like asking the AI to write the code for a house, but instead of giving you a blueprint, it gives you a pile of bricks and a vague idea of a "home." The code it writes usually breaks, has errors, or looks nothing like the picture it was supposed to make.
Templates (The Copy-Paste Job): If you have a "Business Card" template, can the AI recognize that a new design is just a variation of that same template, even if the colors and photos changed?
- The Result: The AI is surprisingly good at this if you give it a multiple-choice list. But if you ask it to figure it out on its own, it gets confused. It's like a student who can pick the right answer on a test but can't explain why it's right.
Animation (The Movie): Can the AI understand a short video animation and tell you which part moves first, or can it create a video where a specific button bounces when you click it?
- The Result: This is the hardest part. The AI is like a person watching a movie with the sound off and trying to guess the plot. It often gets the order of events wrong or makes the whole scene move instead of just the one button.

The Verdict: "Not Ready for Prime Time"

The researchers tested the smartest AI models available (from Google, OpenAI, and Anthropic). Here is the scorecard:

Mostly Solved (2 tasks): The AI is basically perfect at very simple things, like matching two similar-looking templates or ranking them.
Partially Solved (25 tasks): The AI can do the job, but it's sloppy. It might get the general idea right but mess up the details. In a real job, this is like a contractor who builds a wall but leaves the bricks crooked. You'd have to fix it yourself.
Unsolved (23 tasks): The AI fails completely. It can't count objects, it can't read the fonts, and it can't generate the code for a vector icon. It's like asking a robot to perform heart surgery when it doesn't even know what a heart is.

The Big Takeaway

The paper concludes that AI is currently a "creative assistant" but not a "designer."

The "Vibe" is there: AI is great at brainstorming ideas and making things look "cool."
The "Precision" is missing: AI cannot yet be trusted to handle the technical, precise, and structured parts of design work.

If you ask an AI to "make a flyer," it might give you a pretty picture. But if you ask it to "make a flyer with the exact brand font, the correct legal disclaimer at the bottom, and the logo aligned to the grid," it will likely fail.

The Future: The authors say we need to teach AI to understand the "structure" of design, not just the "look" of it. Until then, human designers are still the only ones who can reliably do the job. The AI is a helpful intern who needs constant supervision, not a senior partner.

1. Problem Statement

While recent advances in Vision-Language Models (VLMs) and diffusion-based image generators have raised expectations for AI-assisted design, existing evaluation benchmarks are insufficient for professional graphic design. Current benchmarks primarily focus on:

Natural-image understanding: Object detection, depth estimation, or generic visual question answering.
Generic text-to-image synthesis: Evaluating aesthetic quality or semantic alignment (e.g., FID, CLIP Score) without regard for structural constraints.

The Gap: Professional graphic design is a high-dimensional, constraint-driven task requiring the integration of spatial layout, typography, vector code, brand identity, and temporal animation. Existing benchmarks fail to capture:

Multi-constraint satisfaction: The need to simultaneously render legible text, respect typographic hierarchies, adhere to grid systems, and maintain brand consistency.
Structured output requirements: The necessity for valid vector code (SVG), precise bounding boxes, and layered compositions rather than just pixel-level similarity.
Context dependence: Design quality is often subjective and tied to specific brand identities or user intents, lacking a single "ground truth" in the same way object recognition does.

2. Methodology

Dataset: LICA (Layered-Composition Dataset)

The benchmark is grounded in the LICA dataset, a large-scale collection of real-world graphic design templates from a commercial platform. Unlike flat raster datasets, LICA preserves full structural metadata:

Layered Structure: Individual components (text, image, vector, groups) with bounding boxes, z-ordering, and styling properties.
Rich Annotations: Typography specs (font, size, weight, color, alignment, curvature), animation properties (motion type, duration, keyframes), and global metadata (canvas ratio, category, user intent).
Sibling Variants: Multiple layouts sharing the same structural theme but differing in content, allowing for template variant evaluation.

Benchmark Structure (GraphicDesignBench - GDB)

GDB comprises 50 tasks organized across five design domains and evaluated under two capability modes: Understanding (perception/reasoning) and Generation (synthesis/editing).

Layout: Spatial reasoning, component detection, z-order prediction, and layout generation (intent-to-layout, inpainting, aspect ratio adaptation).
Typography: Font identification, color/size/weight estimation, curved text detection, and styled text generation/removal.
Infographics (SVG & Vector): SVG code understanding (perceptual/semantic Q&A), bug fixing, optimization, and generation (Text/Image-to-SVG/Lottie).
Template & Design Semantics: Category classification, user intent prediction, and template variant matching/generation.
Animation: Temporal reasoning (keyframe ordering), motion type classification, property extraction, and video generation.

Evaluation Framework

Models Evaluated: Three frontier closed-source model families (Google Gemini, OpenAI GPT/Sora/Veo, Anthropic Claude) covering text, image, and video modalities.
Metrics: A custom taxonomy moving beyond FID/CLIP to include:
- Spatial Accuracy: mIoU, mAP, MAE for coordinates.
- Text Fidelity: OCR accuracy, font family matching, color distance ( $\Delta E$ ).
- Structural Validity: JSON/SVG parseability, compression ratios.
- Perceptual Quality: LPIPS, SSIM, DreamSim.
- Human-Aligned Preference: M-Judge (LLM-based pairwise comparison), NIMA, ImageReward.
Solvability Labels: Tasks are categorized as Mostly Solved (>95% performance), Partially Solved (80–95% or mixed results), or Unsolved (<80% or structural failure).

3. Key Contributions

First Comprehensive Design Benchmark: GDB is the first suite to evaluate the full spectrum of professional design capabilities (layout, typography, vectors, semantics, animation) across both understanding and generation.
Layered-Composition Framework: By utilizing the LICA dataset's structural metadata, GDB enables evaluation settings impossible with flat images, such as layer-aware inpainting, partial layout completion, and template variant generation.
Design-Native Metrics: Introduction of a multi-faceted metric taxonomy specifically for design, prioritizing structural validity, typographic fidelity, and spatial accuracy over generic perceptual scores.
Reproducible Baselines: Establishment of strong baselines using frontier models, revealing consistent failure modes across the field and providing a foundation for future open-source and specialized model development.

4. Key Results

The evaluation of frontier models reveals a stark reality: Only 2 of the 50 tasks are "Mostly Solved," while 23 are "Unsolved."

Layout & Spatial Reasoning (Unsolved):
- Component Detection: Best mAP@0.5 is only 6.4%, orders of magnitude below natural image benchmarks.
- Z-Order & Counting: Models struggle to count elements accurately (MAE ~5.8) and predict layer stacking (Kendall's $\tau$ ~0.57, near random).
- Generation: Intent-to-layout generation is bottlenecked by text fidelity (OCR ~75%), and multi-element layout completion fails to reason about spatial interdependencies.
Typography (Unsolved/Partially Solved):
- Font Recognition: Top-1 accuracy is only 23.7% across 167 families; most rare fonts score zero.
- Color & Size: Color prediction varies wildly ( $\Delta E$ up to 52 for weak models); font size estimation has large errors (MAE ~9px vs ~36px).
- Curved Text: Some models fail to detect curved text entirely (0% detection rate).
SVG & Vector (Partially Solved/Unsolved):
- Semantic vs. Perceptual: Models understand what an SVG depicts (Semantic Q&A ~94%) better than visual properties from code (Perceptual Q&A ~87%).
- Generation: Text-to-SVG generation produces schematic interpretations rather than faithful reproductions. Validity drops significantly when adding image input for Lottie generation.
Template Semantics (Mostly Solved for Simple Tasks):
- Pairwise Matching: Surprisingly, simple non-LLM baselines (Font Jaccard) match or exceed LLMs (97.7% accuracy), suggesting LLMs fail to leverage simple metadata cues.
- Category Classification: Open-vocabulary classification is poor due to label aliasing; constrained prompting improves accuracy to ~78%.
Animation (Unsolved):
- Temporal Reasoning: Keyframe ordering exact match is only 16%.
- Motion Classification: Accuracy is below 13% for all models.
- Generation: Models fail to map component indices to spatial regions, leading to globally applied or hallucinated motions. Text legibility and aspect ratio compliance are major failure points in short-form video generation.

5. Significance and Future Directions

Current State of AI Design: The paper concludes that while high-level semantic understanding is within reach, the gap widens sharply as tasks demand precision, structure, and compositional awareness. Current models are not yet capable of functioning as reliable design collaborators for professional workflows.
Implications: The inability to perform precise spatial reasoning, generate valid vector code, or decompose animations prevents AI from moving beyond "rough ideation" into production-ready asset creation.
Future Work:
- Modeling: Design-specialized pretraining on layered composition data and architectures supporting long, structured outputs (to fix SVG truncation).
- Evaluation: Expanding to open-source models, incorporating large-scale human expert evaluation, and developing structure-aware metrics that go beyond pixel similarity.
- Scope: Broadening the dataset to include culturally specific aesthetics, accessibility requirements, and diverse domain languages (packaging, editorial, etc.).

Conclusion: GraphicDesignBench provides a rigorous, reproducible testbed that exposes the fundamental limitations of current AI in professional design. It establishes that the field must shift from optimizing for "aesthetic plausibility" to mastering "structural fidelity" and "constraint satisfaction" to achieve true design automation.