💻 computer science

The Human Creativity Benchmark

The paper introduces the Human Creativity Benchmark (HCB), which distinguishes between professional convergence on technical correctness and divergence in aesthetic taste to argue that evaluating creative AI requires preserving these distinct signals rather than collapsing them into a single quality metric.

Original authors: Aspen Hopkins, Allison Nulty, Alexandria Minetti, Anoop Pakki, Angad Singh

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aspen Hopkins, Allison Nulty, Alexandria Minetti, Anoop Pakki, Angad Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why "Good" is Complicated

Imagine you are trying to judge a cooking competition.

The "Objective" Judge: This judge checks if the food is actually cooked, if the salt is there, and if the plate isn't broken. Everyone agrees on this. If the steak is raw, it's a fail.
The "Subjective" Judge: This judge asks, "Do I like the flavor?" One person might love spicy food, while another hates it. Neither is wrong; they just have different tastes.

The Problem: Most AI tests today act like they only have the "Objective" judge. They try to force everyone to agree on a single score (e.g., "This AI is 8/10"). The authors of this paper say that in creative work (like making ads, websites, or videos), this approach is broken. It throws away the most important part of the data: the fact that people should disagree on taste.

The Solution: The "Martini" of Creativity

The authors propose a new way to test AI called the Human Creativity Benchmark (HCB). They visualize the creative process as a sideways martini glass (see Figure 1 in the paper):

The Wide Part (Ideation): At the start, you want lots of wild, different ideas. This is where "divergence" (different opinions) is good.
The Narrowing Part (Mockup): You start picking the best ideas and making them look real.
The Stem (Refinement): At the end, you are polishing the final product. Here, you need "convergence" (everyone agreeing) on technical details like spelling, layout, and whether the product actually works.

How They Tested It

Instead of asking, "Is this AI good?", they asked professional creatives (designers, video editors, coders) to evaluate AI outputs in three specific ways:

Prompt Adherence: Did the AI do exactly what it was told? (Like a recipe follower).
Usability: Could this actually be used in a real job? (Is the text readable? Is the button clickable?).
Visual Appeal: Is it pretty? (This is where taste comes in).

They ran this test across 15,000 judgments from experts in five different fields: Landing Pages, Desktop Apps, Ad Images, Brand Images, and Product Videos.

What They Found (The "Aha!" Moments)

1. Agreement vs. Disagreement are different signals

Convergence (Agreement): When the AI makes a mistake (like unreadable text or a broken layout), all the experts agree it's bad. When the AI follows the rules perfectly, they also agree it's good.
Divergence (Disagreement): When the AI is technically correct, experts start to disagree based on taste. One expert might say, "This ad looks like luxury fashion," while another says, "This looks cheap." The paper argues this disagreement isn't a mistake; it's a feature. It means the AI is offering different creative directions.

2. No single AI wins everything
If you just look at an average score, you might think one AI is the "best." But the paper found that no model is the best at everything.

Some models are great at the Ideation phase (coming up with wild, creative concepts).
Other models are better at the Refinement phase (fixing typos and making things look professional).
Analogy: It's like saying a Formula 1 car is "better" than a pickup truck. The F1 car wins on the race track (Refinement), but the pickup truck wins at the construction site (Ideation/Utility). You need the right tool for the specific stage of the job.

3. The "Single Score" Trap
The paper warns that if you collapse all these different opinions into one single number (e.g., "Model X is 4.2 stars"), you lose the most useful information. You stop seeing where the model is reliable (technical correctness) and where it is flexible (creative style).

The Takeaway for Humans and AI

The authors suggest that we shouldn't try to make AI agree on a single definition of "beautiful." Instead, we should:

Use AI models that are reliable when we need technical correctness (convergence).
Use AI models that are steerable when we need creative variety (divergence).

In short: Don't ask AI to be perfect at everything. Ask it to be perfect at the right part of the process. The benchmark helps us figure out which AI is the right "chef" for the specific course of the meal we are cooking.

Technical Summary: The Human Creativity Benchmark

Problem Statement

Current AI evaluation frameworks in creative domains treat evaluator disagreement as noise to be resolved, often collapsing diverse professional judgments into a single "ground truth" or aggregate quality score. The authors argue that this approach is fundamentally flawed for creative work. In professional creative contexts, disagreement often reflects genuine differences in taste, aesthetic direction, and creative intent rather than measurement error. By forcing convergence, existing benchmarks obscure the distinction between dimensions where professionals align on shared best practices (convergence) and dimensions where individual preference legitimately varies (divergence). This reductionism risks producing homogenized AI outputs that fail to support the plurality of creative practice and the specific needs of creatives at different stages of a workflow.

Methodology

The paper introduces The Human Creativity Benchmark (HCB), a dataset and framework designed to operationalize the separation of convergence and divergence in creative evaluation.

Data Collection and Structure

Participants: 28 domain experts from Contra, a network of independent professional creatives, spanning 13 countries.
Domains: Five creative domains were evaluated: Landing Pages, Desktop Applications, Ad Images, Brand Images, and Product Videos.
Workflow Phases: The benchmark mirrors real-world creative workflows across three distinct phases:
1. Ideation: Discovery and exploration of creative direction.
2. Mockup: Actualizing the chosen direction into coherent specifications.
3. Refinement: Targeted tweaks for production readiness and consistency.
Models: 13 frontier generative models were evaluated, including image generators (e.g., GPT-Image-1.5, Flux-2-Pro), video generators (e.g., Veo3.1, Kling-v3.0), and code-generation models (e.g., Claude-Opus-4.6, GPT-5.3-Codex).
Evaluation Metrics: Each model output was assessed via three complementary signals:
1. Pairwise Preferences: Forced-rankings of model outputs against each other, aggregated via a Bradley-Terry model to produce ELO ratings.
2. Scalar Ratings: Likert-scale (1–5) ratings on three specific axes:
  - Prompt Adherence: Faithfulness to the prompt (least subjective).
  - Usability: Functional viability in a professional context.
  - Visual Appeal: Aesthetic interest and polish (most subjective).
3. Qualitative Rationales: Open-ended explanations for choices, analyzed via thematic coding.

Analytical Approach

The study employs statistical measures (Kendall's W for agreement, Krippendorff's $\alpha$ for reliability, Friedman tests for significance) to analyze how evaluator agreement shifts across phases and dimensions. The analysis specifically tests whether models perform consistently across the convergence-divergence spectrum and whether model rankings change when evaluated by phase rather than as a single aggregate.

Key Contributions

The Human Creativity Benchmark (HCB): A public dataset comprising 15,000 professional judgments, including prompt-image pairs, pairwise rankings, scalar ratings, and qualitative rationales across five domains and three workflow phases.
A Convergence-Divergence Framework: A theoretical and operational distinction separating creative evaluation into:
- Convergence: Dimensions where experts align on verifiable criteria (e.g., technical correctness, visual hierarchy).
- Divergence: Dimensions where multiple valid judgments exist based on individual taste (e.g., aesthetic direction, conceptual risk).
Empirical Analysis of Creative Workflows: Evidence demonstrating that creative requirements are not static; they shift significantly from ideation to refinement, and that no single model excels uniformly across all phases or domains.

Results

Separation of Standards and Taste: Evaluation axes are not interchangeable. Evaluator agreement is consistently highest on Prompt Adherence (verifiable criteria) and lowest on Visual Appeal (taste-driven). Usability occupies an intermediate position. For example, in Ad Images, agreement on Prompt Adherence remained high, while Visual Appeal scores varied widely based on individual preference.
Phase-Dependent Requirements: The criteria for a "good" output shift as the workflow progresses.
- In Ideation, evaluators prioritize generative potential and structural direction.
- In Mockup, focus shifts to prompt adherence, grid structure, and color consistency.
- In Refinement, agreement peaks as criteria narrow to production-ready elements like typography and specific edits.
No Universal Leader: No individual model led all three phases in any domain. For instance, in Desktop Apps, Claude 4.6 led in Ideation and Mockup, while GPT 5.3 led in Refinement. In Product Videos, Veo 3.1 led Ideation, but Grok Imagine led Refinement.
Signal Loss in Aggregation: Collapsing these distinct signals into a single quality metric discards actionable information. A model might be highly reliable on convergent axes (prompt fidelity) but poor on divergent axes (aesthetic variety), or vice versa.

Significance and Claims

The paper claims that evaluating creative AI requires preserving both convergence and divergence signals. It argues that:

Disagreement is Signal, not Noise: Professional disagreement in creative domains is often a feature, not a bug, reflecting the inherent subjectivity of taste.
Contextual Optimization: Model developers should optimize for reliability on convergent axes (ensuring technical correctness) while maintaining steerability on divergent axes (allowing for creative variation).
Workflow Alignment: Tools should be selected based on the specific phase of the creative process. A model suitable for ideation may not be suitable for refinement, and vice versa.
Preventing Homogenization: By acknowledging divergence, the field can avoid "mode collapse" where AI systems converge to an average, thereby preserving the plurality of creative voices and styles.

The authors conclude that the future of creative AI evaluation lies in moving beyond single-dimensional rankings toward frameworks that recognize the hybrid nature of creative work—balancing objective standards with subjective preference—and supporting the dynamic needs of creatives throughout their workflow.