Technical Summary: TASTE – A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

Problem Statement

Current text-to-image (T2I) models are increasingly deployed for production-scale graphic design, yet their optimization and evaluation rely heavily on preference datasets derived from photorealistic image generation (e.g., HPSv2, Pick-a-Pic, ImageReward). These datasets typically employ a single "which is better" label per comparison, which collapses distinct design axes such as typography, visual hierarchy, color harmony, and layout fidelity. This creates a misalignment where models are rewarded for photorealistic plausibility rather than specific design constraints. Furthermore, existing design-specific resources (e.g., LICA, Graphic-Design-Bench) focus on structural metadata or task completion but lack human preference signals. There is a critical gap in multi-dimensional, designer-annotated preference data that captures the distinct cognitive processes involved in graphic design evaluation.

Methodology

The TASTE Dataset

The authors introduce TASTE (Typography, Aesthetics, Spatial, Tone, Etc.), a dataset designed to capture sub-dimensional preferences.

• Generators: Four current T2I models (FLUX.2 max, GPT Image 1.5, Nano Banana 2, Seedream 5.0 Lite) evaluated under blind code-names.
• Annotators: Two disjoint cohorts of 5 professional designers each (10 total), recruited via Contra.
• Structure:
  • Two Cohorts: An Aesthetics cohort (focusing on mood, tone, hierarchy, color, typography craft) and a Descriptions cohort (focusing on prompt fidelity: color accuracy, spatial accuracy, text rendering).
  • Criteria: 9 distinct sub-dimensions.
  • Scale: 80 prompts per criterion $\times$ 5 evaluators $\times$ 4 models = 1,600 ratings per criterion.
  • Format: Evaluators perform 6 pairwise comparisons per prompt, aggregated into strict 4-way rankings. Additionally, overall-preference cohorts include hallucination flags (No, Minor, Major).

Signal Validation Framework

Before treating the data as training material, the authors propose a criterion-agnostic diagnostic framework to verify the presence of learnable preference signal, distinguishing it from random noise or factional disagreement. The framework tests against exact iid-uniform null distributions at $(p=4, R=5)$ using three statistics:

1. Kendall's $\tau$: Measures rank correlation between rater pairs.
2. Majority Probability ($p_{max}$): Measures the frequency of decisive winners in pairwise comparisons.
3. Condorcet Cycle Indicator: Detects triadic intransitivity ($A \succ B, B \succ C, C \succ A$) to distinguish between shared-ordering noise and factional disagreement.

Benchmarking and Modeling

• Baseline Benchmark: The authors evaluated 9 pre-trained systems (6 open-weight Vision-Language Models and 3 dedicated T2I scorers like HPSv2.1) against the 5-designer majority.
• Custom Model Training: A small modular preference head was trained on TASTE. The authors explored architectural choices, specifically comparing a standard scalar scoring head (scoring images independently) against a pairwise-difference head that explicitly computes features of the difference between two images ($g(t, i_a, i_b)$).

Key Results

1. Signal Quality and Domain Positioning

• Signal Presence: Every TASTE criterion rejects the random-rater null hypothesis on at least two of the three signal statistics ($p < 10^{-10}$).
• Cross-Domain Comparison: Designer agreement on graphic design falls between subjective taste domains (Sushi, MovieLens) and photo-style image quality (HPSv2).
  • Median Kendall's $\tau$ for TASTE ranges from +0.067 (Color Harmony) to +0.200 (Spatial Accuracy).
  • This is lower than HPSv2-test top-4 (0.267) but higher than the iid-uniform null (0).
• Agreement Structure: Cycle rates are at or below the iid-uniform null, ruling out "factional disagreement" (where raters form incompatible groups). The data supports a model of a weak shared preference order with rater-specific noise.
• Signal Heterogeneity: Signal strength varies significantly. Description-fidelity criteria (Typography, Spatial Accuracy) show roughly twice the signal of aesthetic criteria (Color Harmony).

2. Off-the-Shelf Performance

• Low Agreement: No pre-trained system (VLMs or dedicated scorers) exceeds 0.55 macro agreement with the 5-designer majority.
• Scaling Limits: Within the VLM family, scaling parameters (e.g., Qwen3-VL from 4B to 32B) does not improve accuracy. Instead, it shifts the operating point along a bias-versus-perception frontier: larger models exhibit less position bias but also lower content sensitivity when they do commit, resulting in flat net accuracy.
• Specialization: Dedicated aesthetic scorers (e.g., LAION-Aesthetic-V2) fail significantly on text-fidelity tasks (dropping to 0.411 on typography), while HPSv2.1 remains the strongest baseline at 0.543.

3. Model Training Results

• Ceiling Analysis: The theoretical upper bound for a model imitating the panel is the Leave-One-Out (LOO) accuracy of a single rater against the majority of the other four, calculated at 0.741.
• Performance: A custom pairwise-difference head trained on TASTE achieved 0.611 overall accuracy.
  • This closes roughly half the gap between the random baseline (0.5) and the single-rater ceiling (0.741).
  • The improvement is driven almost entirely by better performance on ambiguous 3-2 splits (0.558 $\to$ 0.602), where the model learns to make confident decisions in cases where human raters disagree.
• Architecture: The pairwise-difference architecture (explicitly modeling $i_a - i_b$) outperformed standard scalar heads, suggesting that joint attention to image differences is necessary for this task.

Significance and Claims

The paper positions TASTE not merely as a benchmark, but as a decision layer for the graphic design generation stack. Its significance lies in three areas:

1. Completing the Stack: TASTE fills the missing "human preference" layer between structural data (LICA) and task benchmarks (Graphic-Design-Bench), enabling the evaluation of what designers prefer rather than just what models can do.
2. Methodological Standard: The authors argue that the proposed three-statistic signal check (Kendall's $\tau$, $p_{max}$, Condorcet cycles) should become standard practice for any new preference dataset. It answers the prior question of whether learnable signal exists before investing in model training.
3. Enabling Controllable Alignment: The multi-dimensional structure allows for criterion-specific ranking and controllable alignment. Practitioners can now train reward models or alignment objectives that optimize for specific dimensions (e.g., prioritizing typography fidelity over color harmony) rather than relying on a single aggregate score.

The authors conclude that while current off-the-shelf systems fail to capture designer preference, the dataset provides a verified source of signal for training specialized, designer-aligned preference models, with substantial headroom remaining to reach the human-imitation ceiling.