Technical Summary: AGC-Bench: Measuring Artificial General Creativity

Problem Statement

Creativity research has long debated whether creative ability is a general capacity (analogous to Spearman's g factor in intelligence) or a collection of domain-specific skills. While Large Language Models (LLMs) are increasingly tasked with creative output, the field lacks a unified infrastructure to evaluate whether a model's creativity generalizes across domains or remains siloed. Existing evaluations are fragmented across hundreds of heterogeneous benchmarks with incompatible scoring protocols, making it empirically intractable to determine if a model that excels at creative writing also possesses strong capabilities in scientific ideation, humor, or problem-solving. Furthermore, the reliance on "LLM-as-a-judge" introduces systematic biases in leniency and severity that obscure true performance differences.

Methodology

1. AGC-Bench Construction

The authors introduce AGC-Bench, a meta-benchmark constructed through a PRISMA-compliant systematic review of AI creativity literature.

• Literature Review: From 3,101 candidate papers (2018–2025), the team identified 497 unique benchmarks.
• Onboarding Pipeline: An agentic workflow converted 78 of these benchmarks into standardized, runnable HELM-style scenarios. This included 67 text-only datasets and 11 multimodal scenarios.
• Taxonomy: The text-only datasets were organized into six domains grounded in human creativity research: Brainstorming, Problem Solving, STEM, Story/Narrative, Figurative Language, and Humor.
• Evaluation Scale: 83 frontier LLMs (proprietary and open-weight) were evaluated across these datasets. Each (model, dataset) pair was scored on $n=50$ items, with rank stability confirmed via bootstrap.

2. Scoring and Calibration (Judge Response Theory)

To address bias in automated scoring, the authors applied Judge Response Theory (JRT):

• Three-Judge Panel: A planned-missing "2-of-3" design was used, where each item was rated by two of three frontier LLM judges (Gemini-3-Flash, Grok-4.1-Fast, GPT-4.1-mini).
• Bayesian Calibration: A Bayesian Graded Response Model (GRM) was fitted to recover per-judge severity and discrimination parameters, correcting for systematic leniency or harshness.
• AGC-Judge: A Qwen3-30B-A3B-Instruct model was fine-tuned on 48,299 JRT-corrected ratings to create AGC-Judge, an open-weight scoring model that replicates the three-judge ensemble at high accuracy ($\rho \ge 0.97$) while predicting ratings on unseen benchmarks with high fidelity ($\rho = 0.83$).

3. Psychometric Analysis

The study employed factor analysis on the 83 models $\times$ 6 domains matrix to test for a general creativity factor.

• External Validity: The benchmark's construct validity was anchored against a pure fluid reasoning ($G_f$) probe (Lewis-Mitchell counterfactual letter-string analogies), a human-rated creativity battery (MuCE), and within-model interventions (prompting models to "be creative" vs. "be effective," and toggling reasoning modes).
• Human Comparison: A subset of five tasks (AGC-Human) allowed for direct comparison between 224 human participants and 125 LLM evaluations.

Key Results

1. Existence of a General Creativity Factor (c)

Factor analysis across the 83 models revealed a single, dominant creativity factor, denoted as $c$, which explains 81.5% of the variance across the six domains (Cronbach's $\alpha = 0.96$).

• Separability: While $c$ correlates with fluid reasoning ($\rho = 0.55$) and general knowledge ($\rho = 0.62$), it remains statistically separable from both. It is also separable from model scale, though total parameter count correlates moderately ($\rho = 0.69$).
• Robustness: The factor persists after partialling out fluid reasoning, across different scoring families (formula-based vs. LLM-judge), and under model subsampling.

2. Domain-Specific Performance Variations

While a general factor exists, performance is not uniform.

• Leaderboard: Proprietary frontier models (e.g., Claude Opus 4.7, GPT-5.4/5.5) lead the overall composite, with open-weight models (e.g., GLM-5.1) close behind.
• Dissociations: Specific models show distinct strengths. For instance, kimi-k2.5 tops Problem Solving but does not rank in the top 5 for other domains, while claude-opus-4.6-fast leads Figurative Language significantly.

3. Sensitivity to Creativity-Specific Prompts

Intervention experiments demonstrated that the benchmark tracks creativity specifically rather than general reasoning:

• Prompting models to "be creative" resulted in a large performance boost ($d_z = +1.40$), whereas enabling reasoning capabilities produced a much smaller shift ($d_z = +0.34$).

4. Human vs. LLM Creativity

On the paired-human subset:

• Domain Generality: Creativity is significantly more domain-general in LLMs ($\alpha = 0.64$) than in humans ($\alpha = 0.42$).
• Performance Ceiling: The top human performer still leads the top LLM by 0.12 standard deviations, though frontier models are approaching human-level performance on ideation and creative writing tasks.

Key Contributions

1. AGC-Bench: A large-scale meta-benchmark comprising 78 datasets (67 text-only, 11 multimodal) unified under a single evaluation harness, representing an order-of-magnitude increase in scope over prior work.
2. AGC-Judge: An open-weight, fine-tuned scoring model that reproduces the consensus of a three-judge frontier panel at a fraction of the cost, enabling scalable and reproducible creativity evaluation.
3. Psychometric Infrastructure: A reproducible pipeline combining PRISMA systematic review with agentic onboarding and JRT calibration, establishing a formal test for the "Artificial General Creativity" hypothesis.
4. Empirical Evidence: The first large-scale demonstration that LLM creativity is largely unidimensional (a general $c$ factor) yet distinct from general intelligence and reasoning.

Significance

The paper provides the first empirical test of whether creative ability in LLMs is a general capacity or a set of domain-specific skills. By recovering a strong general creativity factor ($c$), the authors suggest that creativity in AI, much like intelligence, may be a transferable capability that scales with model size and training, rather than a purely emergent property of specific tasks. The release of AGC-Bench, the onboarding harness, and AGC-Judge offers the community a standardized infrastructure to move beyond fragmented, single-domain evaluations toward a holistic understanding of AI creativity. The authors note that while current models approach human performance, the top human still leads, and the benchmark currently lacks a theoretical ceiling, serving as a relative measure against the current frontier.