Technical Summary: Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity

Problem Statement

While Large Language Models (LLMs) are increasingly deployed to evaluate creative ideas in both individual and organizational contexts, empirical evidence regarding their alignment with human judgments remains mixed. Existing literature offers conflicting findings: some studies suggest LLMs can approximate human creativity judgments, while others indicate weak alignment or significant model-specific variations. The core problem addressed by this research is a lack of theoretical and empirical understanding regarding which standards LLMs apply when evaluating creativity, where these standards converge with or diverge from human standards, and how these differences manifest in downstream evaluation outcomes. The authors posit that agreement is not a binary capability but depends on the specific evidence a judgment demands and the standards each model prioritizes.

Methodology

The research employs a three-study design to map LLM evaluation standards and examine their downstream consequences.

Study 1: Mapping Evaluation Standards

• Objective: To identify the creativity evaluation standards used by six state-of-the-art LLMs (GPT-4.1 mini, Grok 4 Fast, DeepSeek V3, ERNIE 3.5-8K, Claude Haiku 4.5, and Gemini 2.5 Flash-Lite) and compare them to established human standards.
• Procedure: Using an established measure of human creativity evaluation (identifying 26 distinct standards across seven dimensions), the authors prompted each LLM to rate the importance of each standard.
• Analysis: Standards were deemed "important" if their 95% confidence intervals lay above the scale midpoint. The authors analyzed the aggregate LLM standards and applied Cultural Consensus Theory to determine if models shared a universal standard or exhibited model-specific patterns.

Study 2: Downstream Consequences of Convergence

• Objective: To test whether the breadth of LLM standards correlates with the accuracy of their creativity ratings relative to humans.
• Procedure: 293 human participants generated 1,103 creative ideas across three tasks (two in-basket business scenarios and one environmental task). Three trained human evaluators and the six LLMs independently rated these ideas.
• Analysis: The authors calculated Pearson correlations between aggregate and individual LLM ratings and human ratings (derived via the Consensual Assessment Technique). They tested whether models with broader standards (more aligned with human standards) showed higher correlation with human judgments.

Study 3: Downstream Consequences of Divergence

• Objective: To investigate how the divergence in the "contextual" dimension (where LLMs and humans differ most) affects creativity ratings.
• Procedure: A 2 (Product Description: product-only vs. product-with-context) × 2 (Evaluator Source: human vs. LLM) experimental design was used with 15 Kickstarter products. Contextual information (e.g., brand prestige, market positioning) was added to product descriptions.
• Analysis: Multilevel moderation analyses tested whether contextual information shifted creativity ratings differently for humans versus LLMs.

Key Results

1. LLMs Rely on a Narrower, Idea-Centered Subset of Standards

• Partial Overlap: While humans utilize all 26 identified creativity standards, the general LLM standard incorporates only 17 (65.38% overlap).
• Convergence: LLMs converge most strongly with humans on the Novelty dimension (all novelty standards were included) and the Hedonic dimension.
• Divergence: The clearest divergence occurs in the Contextual dimension. Humans heavily weigh social, market, and reputational factors (e.g., "name brand," "mass market," "fashionability"), whereas LLMs generally exclude these, treating them as weak evidence of creativity. Only "credibility" was marginally included.
• Model Variability: LLMs do not share a single universal standard. Cultural Consensus Theory revealed six distinct clusters (one per model).
  • Broad Standards: GPT and Grok incorporated 20 standards each.
  • Moderate Standards: DeepSeek and ERNIE incorporated 14 standards.
  • Narrow Standards: Claude and Gemini incorporated 12 and 11 standards, respectively, remaining tightly anchored to novelty.

2. Standard Breadth Predicts Alignment with Human Judgments (Study 2)

• Aggregate Alignment: There was a moderate correlation ($r = 0.50$) between aggregate LLM ratings and human ratings.
• Model Hierarchy: Alignment strength was directly correlated with the breadth of the model's standards.
  • GPT and Grok (broadest) showed the strongest alignment ($r \approx 0.50$).
  • DeepSeek and ERNIE (moderate) showed intermediate alignment ($r \approx 0.46$).
  • Claude and Gemini (narrowest) showed the weakest alignment ($r \approx 0.41$).
• Conclusion: Models that preserve a broader range of human standards better distinguish between ideas humans judge as more or less creative.

3. Contextual Information Drives Asymmetric Evaluation (Study 3)

• Human Sensitivity: Providing contextual information (e.g., brand prestige) significantly increased human creativity ratings ($b = 0.18, p < 0.001$).
• LLM Insensitivity: The same contextual information had no significant effect on LLM ratings ($b = 0.01, p = 0.367$).
• Specificity: This insensitivity was most pronounced for "name brand," "mass market," and "fashionability" variants—the exact standards LLMs downweighted in Study 1.
• Conclusion: LLMs are less sensitive to contextual evidence that substantially shapes human creativity judgments, leading to systematic deviations when such information is present.

Significance and Contributions

Theoretical Contributions

• Explaining Mixed Evidence: The paper resolves inconsistencies in prior literature by demonstrating that LLM-human alignment is not a monolithic "yes/no" capability but depends on the specific evaluation standards required. Alignment is high when tasks emphasize intrinsic qualities (novelty) but low when they require contextual judgment.
• Revisiting Creativity Theory: The findings reveal a discrepancy between formal creativity theories (which emphasize both novelty and usefulness) and LLM standards. LLMs disproportionately weight novelty, likely because it is more explicitly and consistently verbalized in their training data, whereas standards like usefulness or social approval are applied more tacitly by humans and thus less visible in textual corpora.
• Model Heterogeneity: The research challenges the treatment of "LLMs" as a single category, showing that different models apply distinct, model-specific standards that significantly alter which ideas are recognized as creative.

Practical Implications

• Context-Dependent Utility: LLMs are effective evaluators when the goal is to assess intrinsic idea quality (e.g., novelty, conceptual promise) while minimizing bias from social or reputational signals (e.g., early-stage scientific proposal screening).
• Limitations in Context-Rich Settings: In domains where market fit, social norms, or stakeholder credibility are integral to creativity (e.g., product design), LLM evaluations may be incomplete or misleading unless supplemented by human judgment.
• Selection Strategy: Organizations must treat the choice of an LLM evaluator as a consequential decision, selecting models whose specific standard breadth aligns with the evaluation goals.

Limitations and Future Directions
The authors acknowledge that Study 1 relied on self-reported importance ratings, which may not perfectly mirror behavioral standards in all contexts. They also note that the study focused on six proprietary models, leaving open questions about open-source, domain-specialized, or multimodal models. Finally, the paper raises the possibility of a feedback loop where repeated interaction with LLM evaluators could eventually shift human evaluation standards, a phenomenon requiring future investigation.