Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective

This paper introduces AesEval-Bench, a comprehensive benchmark and training dataset designed to systematically evaluate and enhance Vision Language Models' ability to assess graphic design aesthetics across multiple dimensions, tasks, and indicators.

The Gist

Imagine you have a very smart robot assistant that can see pictures and read text. You've taught it to identify cats, read street signs, and even write poems. But now, you ask it a much trickier question: "Is this poster beautiful? And if not, exactly what is wrong with it?"

This is the challenge tackled in the paper "Can Vision–Language Models Assess Graphic Design Aesthetics?" by Arctanx An and colleagues.

Here is the story of their work, broken down into simple concepts with some creative analogies.

1. The Problem: The Robot Can't "Get" Art

Currently, AI models (called Vision-Language Models or VLMs) are like super-smart tourists. They can tell you, "That's a picture of a beach," or "The text says 'Sale'." But they struggle to be art critics.

When you ask a human designer, "Why does this flyer look messy?" they might say, "The title is too small, the colors clash, and the text is squished against the edge."
When you ask a standard AI, it often just guesses "Yes" or "No" without knowing why, or it gives a vague answer like "It looks okay."

The researchers found three big gaps:

• The Tests Were Too Easy: Previous tests only asked simple questions about photos, ignoring specific design rules like font choice or layout.
• No Fair Comparisons: Nobody had systematically tested all the different AI models to see which one is actually the best "design critic."
• No Training Manual: There was no "textbook" to teach these AIs how to spot bad design.

2. The Solution: Building "AesEval-Bench" (The Design School)

To fix this, the team built a new testing ground called AesEval-Bench. Think of this as a final exam for AI art critics.

Instead of just asking "Is this pretty?", the exam is broken down into three levels of difficulty, covering four main subjects: Typography (Fonts), Layout, Color, and Graphics.

• Level 1: The Gut Check (Aesthetic Judgment)
  • The Question: "Is this design good or bad?" (Yes/No).
  • The Analogy: Like a judge at a talent show giving a simple thumbs up or down.
• Level 2: The Detective Work (Region Selection)
  • The Question: "Here are four parts of the image. Which one looks bad?"
  • The Analogy: Like a game of "Where's Waldo?" but you have to find the ugly Waldo.
• Level 3: The Surgeon's Precision (Precise Localization)
  • The Question: "Draw a box around the exact part that is ugly."
  • The Analogy: Like a surgeon pointing a laser at the exact spot that needs to be fixed.

They created 4,500 of these test cases using professional designs that they intentionally "ruined" (by shifting text, changing colors, or making fonts blurry) to see if the AI could spot the mistakes.

3. The Results: The AI is Still a Rookie

The researchers put 10 different AI models (from big companies like OpenAI and Google, plus open-source ones) through this exam.

• The Verdict: Even the smartest AIs struggled. They got the "Gut Check" right about 70% of the time, but when it came to pinpointing the exact bad spot (Level 3), they were barely better than random guessing.
• The Surprise: The "Reasoning" models (AIs designed to think step-by-step like a human logician) didn't do much better than the standard ones. It turns out, just "thinking harder" doesn't help if you don't know the rules of design.
• The Gap: There is a huge difference between an AI that knows what a "dog" looks like and an AI that knows what "good typography" looks like.

4. The Fix: Teaching the AI with a "Design Tutor"

Since the AIs were failing, the team decided to build a training dataset (AesEval-Train) to teach them properly.

They used two clever tricks:

1. Human-Guided Labeling: Instead of hiring humans to grade thousands of designs (which is expensive), they used a few human examples to teach a powerful AI how to grade the rest. It's like showing a student a few solved math problems so they can do the rest of the homework.
2. Indicator-Grounded Reasoning: This is the secret sauce. Usually, an AI might say, "The colors are bad." But the researchers forced the AI to say, "The colors are bad because the red text on the blue background (pointing to specific coordinates) creates a clash."
   • The Analogy: Instead of just saying "This car is broken," the AI learns to say, "The engine is broken because the spark plug in cylinder 3 is missing." It ties the abstract concept (bad design) to the physical reality (the specific pixels).

5. The Outcome: A New Level of Understanding

After training the AI with this new "Design Tutor" dataset, the results improved dramatically.

• The AI got much better at spotting bad designs.
• It got significantly better at drawing the box around the bad part.
• Most importantly, the AI started explaining why something was wrong by pointing to the specific area, just like a human designer would.

Summary

This paper is like a report card for the future of AI design tools.

• The Bad News: Current AI isn't ready to replace human designers yet; it doesn't truly "get" aesthetics.
• The Good News: The researchers built the first comprehensive test and the first effective training method to teach AI how to critique design.

By giving AI a structured way to learn (connecting abstract rules to concrete pixels), they have paved the way for future tools that can not only make designs but also critique and improve them with human-like insight.

Technical Summary

1. Problem Statement

While Vision-Language Models (VLMs) have advanced significantly in tasks like image captioning and visual question answering, their ability to assess the aesthetic quality of graphic design remains underexplored. Existing research faces three critical limitations:

1. Inadequate Benchmarks: Prior benchmarks are often restricted to natural photos (ignoring design-specific factors like typography) or cover only narrow subsets of design principles. Evaluation protocols are often qualitative (free-form descriptions) or coarse (holistic scores), lacking the granularity to identify where aesthetic failures occur.
2. Lack of Systematic Comparison: There is no comprehensive evaluation comparing proprietary, open-source, and reasoning-augmented VLMs specifically for design aesthetics.
3. Data Scarcity: There is a lack of high-quality training data to fine-tune VLMs for this specific domain, particularly data that links abstract aesthetic indicators to concrete visual regions.

The core research question is: Can VLMs evaluate graphic design aesthetics in a manner comparable to humans, and if not, how can they be improved?

2. Methodology

The authors propose a three-pronged approach: a new benchmark (AesEval-Bench), a systematic evaluation of existing models, and a novel training dataset construction pipeline (AesEval-Train).

A. AesEval-Bench (The Benchmark)

The benchmark is structured around four critical dimensions and twelve specific indicators derived from design literature and expert consultation:

• Dimensions: Typography, Layout, Color, and Graphics.
• Indicators: Includes hierarchy, legibility, balance, layering, whitespace, alignment, harmony, contrast, appeal, psychology, quality, and relevance.

The benchmark defines three quantifiable tasks for each indicator:

1. Aesthetic Judgment: A binary classification (Yes/No) on whether a design is aesthetically pleasing.
2. Region Selection: A multiple-choice task where the model must select the specific region (from 4 candidates) containing an aesthetic flaw.
3. Precise Localization: A regression task requiring the model to predict the exact bounding box (bbox) coordinates of the flawed region.

Data Construction:

• Source: 1,200 professional graphic designs from the Crello dataset.
• Perturbation: Controlled perturbations (e.g., shifting elements, altering fonts/colors) are applied to create "flawed" variations.
• Annotation: Human annotators verify flaws. A "Human-in-the-Loop" process ensures consensus via majority voting.
• Scale: The final benchmark comprises ~4,500 question-answer pairs (3 tasks × 12 indicators × 1,500 designs).

B. Systematic Evaluation

The authors evaluated 10 VLMs, including:

• Non-reasoning models: LLaVA, Qwen-VL (7B/32B/72B), Intern-VL, GPT-4o, GPT-5.
• Reasoning-augmented models: GPT-o1, GPT-o3, Gemini-2.5-Pro.
• Expert models: AesExpert, UNIAA-LLaVA (specialized for image aesthetics).

Metrics: Accuracy for judgment/selection; Intersection over Union (IoU) for localization.

C. AesEval-Train (Training Dataset & Strategy)

To address performance gaps, the authors constructed a training dataset using two novel strategies:

1. Human-Guided VLM Labeling: Instead of expensive full human annotation, a small set of human labels serves as in-context examples to guide powerful VLMs in generating task labels at scale.
2. Indicator-Grounded Reasoning: Unlike generic reasoning, this approach explicitly ties abstract indicators (e.g., "hierarchy") to concrete design regions. The reasoning path includes:
   • Bounding box coordinates of the relevant region.
   • Textual explanations linking the visual evidence to the specific indicator.

The models are fine-tuned using the task input, the reasoning path, and the task label as supervision.

3. Key Results

Benchmark Evaluation Findings

• Performance Gaps: Even state-of-the-art proprietary models (e.g., GPT-5) struggle, achieving only ~72.5% accuracy on aesthetic judgment. Open-source models generally lag behind, though larger variants (72B) outperform smaller ones (7B).
• Reasoning Limitations: Surprisingly, reasoning-augmented models (e.g., GPT-o1) showed no clear advantage over their non-reasoning counterparts. Generic chain-of-thought reasoning does not inherently improve aesthetic assessment.
• Domain Specificity: Models specialized in general image aesthetics (AesExpert) performed worse than general VLMs, highlighting that graphic design aesthetics differ significantly from general image aesthetics (requiring understanding of typography, layout, etc.).
• Task Difficulty: Performance drops significantly from judgment to region selection, and is lowest for precise localization (IoU < 0.20 for the best models), indicating models struggle to pinpoint exact flaw locations.
• Input Sensitivity: Removing design images causes the largest performance drop. Metadata (JSON) is crucial for localization tasks, while detailed indicator explanations are vital for subjective indicators (e.g., psychology).

Fine-Tuning Results

• Significant Gains: Fine-tuning Qwen-VL-7B with AesEval-Train yielded substantial improvements:
  • Aesthetic Judgment: +5.97% accuracy (surpassing the 72B base model).
  • Region Selection: +2.70% accuracy.
  • Precise Localization: +17.17% IoU (outperforming GPT-5 despite having fewer parameters).
• Ablation Study:
  • Removing the Reasoning Path reduced performance, proving that linking indicators to specific regions is critical.
  • Removing Positive Samples (training only on flawed designs) caused a collapse in performance, emphasizing the need for balanced data.

4. Key Contributions

1. AesEval-Bench: The first comprehensive benchmark for graphic design aesthetics, covering 4 dimensions, 12 indicators, and 3 fully quantifiable tasks (Judgment, Selection, Localization).
2. Systematic Evaluation: A rigorous comparison revealing that current VLMs (including reasoning-enhanced ones) have significant gaps in design assessment and that larger models do not always solve the problem without domain adaptation.
3. Training Framework: A novel pipeline for constructing training data using Human-Guided VLM Labeling and Indicator-Grounded Reasoning, which effectively bridges the gap between abstract design principles and concrete visual evidence.
4. Performance Validation: Demonstrated that fine-tuning with this specific data strategy allows smaller models to outperform much larger, untrained proprietary models.

5. Significance

This work establishes a foundational framework for automated graphic design critique. By moving beyond holistic scoring to region-specific, indicator-grounded evaluation, it enables:

• Assistive Design Tools: Providing designers with specific, actionable feedback on where and why a design fails.
• Generative AI Feedback Loops: Enabling iterative refinement of AI-generated designs without constant human intervention.
• Theoretical Insight: It challenges the assumption that "more reasoning" or "larger models" automatically solve complex aesthetic tasks, showing that domain-specific grounding (linking concepts to pixels) is the key to success.

The paper concludes that while VLMs are not yet human-level in design assessment, the proposed benchmark and training methodology provide a clear path forward for developing specialized, high-performance aesthetic evaluators.