Technical Summary: Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in reasoning and generation, their creative potential remains difficult to quantify systematically. Existing creativity evaluation frameworks suffer from three primary limitations:

1. Task-Specific Coupling: Most metrics are tightly bound to specific domains (e.g., coding, math), embedding strong domain assumptions that limit generalizability.
2. Scalability Issues: Reliance on curated answer sets, hand-crafted rubrics, and extensive human annotation makes assessment subjective, resource-intensive, and unscalable.
3. Lack of Reference-Free Metrics: Current automated methods often fail to distinguish between genuine conceptual diversity and superficial surface-level variations (e.g., paraphrasing).

Consequently, the field lacks a standardized, automated, and domain-agnostic method to evaluate LLM creativity across diverse open-ended tasks.

Methodology

The authors propose a fully automated, domain-general framework that decouples the measurement apparatus from the creative task itself. The framework operationalizes creativity based on the cognitive science distinction between divergent and convergent thinking (Guilford, 1950).

1. Divergent Creativity: Semantic Entropy (SE)

To measure the breadth and novelty of ideas (divergent thinking), the authors adapt Semantic Entropy, originally proposed for hallucination detection in single-answer QA (Farquhar et al., 2024).

• Concept: In open-ended tasks, probability mass distributed across semantically distinct ideas reflects a model's exploration of alternative solution paths rather than error.
• Implementation:
  • For a given query, the model samples $n=10$ candidate solutions.
  • These generations are clustered based on semantic equivalence using bi-directional entailment.
  • SE is computed as the entropy of the probability distribution over these semantic classes: $H(x) = -\sum P(C_i|x) \log P(C_i|x)$.
  • This metric operates at the level of meaning, avoiding inflation from superficial paraphrasing that affects word-level diversity metrics (e.g., Distinct-n, Self-BLEU).

2. Convergent Creativity: Retrieval-Based Multi-Agent Judge

To measure task fulfilment, feasibility, and context-appropriateness (convergent thinking), the authors introduce a novel retrieval-based multi-agent judging framework.

• Motivation: Traditional multi-agent discussion frameworks (e.g., ChatEval) append full discussion histories at every turn, causing token costs to grow prohibitively for large-scale benchmarking.
• Mechanism:
  • Specialized agents (Problem, Solution, Criterion) analyze tasks and record insights as "fragments."
  • Instead of replaying the full history, agents embed fragments in a vector database.
  • At each turn, agents retrieve only the $k$ most relevant prior fragments via cosine similarity.
  • This design maintains multi-agent deliberation while reducing token usage by approximately 63% compared to ChatEval, enabling scalable evaluation without sacrificing accuracy.

3. Experimental Setup

The framework was validated across three qualitatively distinct domains:

• MacGyver: Unconventional physical problem-solving (evaluating feasibility, safety, effectiveness).
• HypoGen: Scientific research ideation (evaluating feasibility, scientific accuracy, relevance).
• BookMIA: Creative writing (evaluating coherence, realism, plot completion).

The evaluation protocol involves a two-stage process: sampling multiple next-step continuations to compute SE (divergent), followed by greedy decoding to construct a final solution for the multi-agent judge (convergent).

Key Results

Divergent Creativity Findings

• Validity: Semantic Entropy shows the highest agreement with human judgments of idea breadth ($\kappa = 0.56$) compared to cosine similarity, Self-BLEU, and Distinct-n metrics.
• Correlations: Step-level SE correlates positively with the number of unique semantic clusters and negatively with pairwise cosine similarity, confirming it captures genuine conceptual diversity.
• Scaling Behavior: Contrary to convergent metrics, divergent creativity does not scale monotonically with model size or recency. Larger and more recent models (e.g., LLaMA 3.3, GPT-4o) often exhibit stable or even decreased SE compared to smaller counterparts. The authors suggest current training paradigms may prioritize solution correctness over creative exploration.

Convergent Creativity Findings

• Performance: The retrieval-based multi-agent framework achieves accuracy comparable to human annotators (e.g., 84.7% vs. 82.7% for GPT-4o vs. Human on MacGyver) while being significantly more efficient.
• Scaling Behavior: Convergent creativity (task fulfilment) does scale with model size, recency, and the inclusion of reasoning capabilities. Models like GPT-4o and LLaMA 3.1 70B consistently outperform smaller or older models.

Relationship Between Dimensions

• Empirical Separability: There is a weak correlation between Semantic Entropy (divergent) and convergent creativity metrics. High diversity does not inherently lead to poor task fulfilment, nor does high task fulfilment guarantee high diversity.
• Interpretation: This suggests that "creative breadth" and "task correctness" are distinct dimensions that may be independently optimizable.

Significance and Claims

The paper claims to establish a reproducible and generalizable standard for automated LLM creativity evaluation. Its primary contributions are:

1. A Reference-Free Metric: Introducing Semantic Entropy as a robust, task-agnostic measure of divergent creativity that aligns with human judgments of novelty and flexibility.
2. Scalable Evaluation: Developing a retrieval-based multi-agent framework that reduces computational costs by over 60% while maintaining high evaluation fidelity, making large-scale benchmarking feasible.
3. Insight into Model Behavior: Providing empirical evidence that current LLM training paradigms optimize for structured reasoning and correctness (convergent) but do not necessarily expand creative exploration (divergent).

The authors conclude that by decoupling measurement from the creative task, their framework provides a foundation for studying how architectural and training choices shape creative behavior, paving the way for scalable benchmarking and the development of "creative AI."

Limitations

The authors acknowledge that the framework relies on LLM-based judgments, which inherently carry the subjective interpretations and prior knowledge encoded in the models. This may lead to unconventional solutions being incorrectly deemed infeasible. Additionally, the computational overhead of generating and clustering multiple outputs per task remains a constraint for extremely large datasets.