Technical Summary: Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in tasks requiring verbal creativity, such as storytelling and idea generation. However, current benchmarks for evaluating LLM creativity lack a solid grounding in human creativity theory, which defines creativity as the combination of novelty and appropriateness.

The standard metric for assessing divergent thinking, the Divergent Association Task (DAT), measures creativity solely based on the semantic distance between generated words (novelty). The authors argue that this novelty-only approach is flawed for LLMs because:

• It ignores appropriateness, a core component of human creativity.
• It allows stochastic or random outputs to achieve high scores, conflating randomness with creativity.
• Previous studies showed that non-creative baselines (e.g., random sampling) can outperform state-of-the-art LLMs on DAT, invalidating the metric for model evaluation.

Consequently, there is a need for an evaluation framework that objectively measures creativity while adhering to the dual criteria of novelty and contextual appropriateness.

2. Methodology

Critique of the Divergent Association Task (DAT)

The authors first re-evaluated the standard DAT using a range of state-of-the-art (SotA) LLMs.

• Task: Models were prompted to generate 10 nouns as different from each other as possible.
• Scoring: Novelty was calculated as the average pairwise semantic distance between the first seven valid words.
• Embedding Shift: While standard DAT uses GloVe embeddings, the authors utilized Sentence BERT (SBERT) to mitigate frequency artifacts that inflate novelty scores for rare words.
• Baselines: Two non-creative baselines were introduced:
  1. Task-agnostic: Randomly sampling 10 nouns from WordNet.
  2. Task-aware: Prompting models to explicitly maximize the DAT score (optimizing the test objective directly).

Introduction of Conditional Divergent Association Task (CDAT)

To address DAT's limitations, the authors propose CDAT, grounded in the psychological definition of creativity requiring both novelty and appropriateness (Heinen and Johnson, 2018).

• Task Specification: Given a specific cue noun, the model must generate 10 mutually dissimilar nouns that are semantically associated with the cue.
• Evaluation Protocol:
  • Novelty: Measured as the average pairwise semantic distance among the generated words (identical to DAT scoring).
  • Appropriateness: Measured as the average semantic similarity between the generated words and the cue word.
  • Appropriateness Gate: A filtering mechanism where a model's generated words must statistically exceed the appropriateness of a random baseline (task-agnostic) to be considered valid. Novelty is only scored if the model passes this gate.
  • Final Score: The primary metric is the mean novelty score, conditional on passing the appropriateness gate.
• Visualization: The authors analyze the trade-off between novelty and appropriateness using a 2D landscape and a Pareto front to identify non-dominated models that offer optimal balances.

Experimental Setup

• Models: A diverse set of proprietary and open-source SotA LLMs of varying sizes and families.
• Conditions: Evaluations were conducted at three temperature settings (0.5, 1.0, 1.5) and a deterministic setting (0.0).
• Cues: 539 unique cue nouns derived from the Brown corpus.
• Baselines: Included Random (task-agnostic) and Common (task-aware, prompting for maximum association) baselines for 2D diagnostic analysis.

3. Key Results

DAT Validity Failure

Under the SBERT-based evaluation, no LLM family exceeded the task-agnostic (random) baseline in DAT scores at any temperature. Furthermore, the task-aware baseline (which "cheats" by optimizing for the score) consistently outperformed standard model generations. This confirms that DAT is sensitive to random sampling and prompt familiarity rather than isolating genuine creative ability.

CDAT Findings

When evaluated under CDAT, distinct patterns emerged regarding model families and the novelty-appropriateness trade-off:

1. Gate Passing: All models passed the appropriateness gate, validating that novelty can be measured conditionally on contextual alignment.
2. Model Family Divergence:
   • Smaller/Basic Families: Generally achieved higher CDAT scores, occupying regions of higher novelty but lower appropriateness.
   • Advanced/Larger Families: Tended to favor higher appropriateness at the cost of lower novelty, resulting in more conservative CDAT scores.
3. Pareto Frontier: Both strategies (high novelty/low appropriateness vs. balanced) appeared on the non-dominated Pareto frontier, indicating that no single model family simultaneously maximizes both dimensions.
4. Temperature Effects: Temperature effects were mixed and family-specific, with no consistent trend across all models.

4. Key Contributions

• Critical Evaluation of DAT: The paper demonstrates that the standard DAT is invalid for LLM creativity evaluation because non-creative baselines outperform SotA models, and the metric fails to distinguish between randomness and creativity.
• Proposal of CDAT: Introduction of the Conditional Divergent Association Task, which operationalizes creativity as novelty conditional on contextual appropriateness, separating noise from creative output.
• Nuanced Profiling: CDAT reveals systematic differences in creativity profiles across model families, showing that smaller models often exhibit higher raw novelty, while larger, aligned models prioritize appropriateness, likely due to training and safety alignment shifts.

5. Significance and Claims

The authors claim that CDAT provides a simple, objective, and human-interpretable metric that aligns with established psychological theories of creativity. By grounding evaluation in both novelty and appropriateness, CDAT avoids the pitfalls of novelty-only metrics.

The paper posits that the observed trade-off—where advanced models sacrifice novelty for appropriateness—is likely a result of training and alignment processes that shift models toward conservative, predictable outputs. However, the authors maintain a modest stance, noting that this is a correlational interpretation requiring further causal ablation studies (e.g., on alignment objectives and decoding policies) to confirm.

The work is presented as a pragmatic operationalization for quantifying a minimal component of divergent verbal creativity, specifically targeting crystallized, text-based knowledge tasks where LLMs are most frequently deployed. It is not claimed to be a universal predictor of all forms of human creativity but serves as a necessary step toward more rigorous, theory-grounded benchmarks for machine creativity.