Technical Summary: Rethinking Creativity Evaluation

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in structured tasks like logical reasoning and question answering, their ability to generate creative content remains limited and difficult to evaluate. Creativity is inherently multidimensional and domain-specific, yet there is a lack of reliable, automated metrics that can effectively benchmark LLM performance across diverse creative domains. Existing evaluation methods often fail to align with human perceptions of creativity, leading to inconsistencies in how "creative" outputs are identified and measured. This paper addresses the critical question: How can we select evaluation metrics that accurately reflect human-perceived creativity across different domains?

Methodology

The authors conducted a comprehensive analysis of four representative creativity metrics across three distinct creative domains:

1. Domains:

   • Creative Writing: Artistic and aesthetic content generation (e.g., movie plots, poems).
   • Unconventional Problem-Solving: Functional thinking involving breaking cognitive biases (e.g., the MacGyver task).
   • Research Ideation: Logical synthesis of existing facts to generate novel scientific ideas (e.g., ICLR paper submissions).

2. Datasets:
   For each domain, the authors curated datasets containing paired examples of "creative" and "uncreative" outputs, labeled based on human judgment.

   • Creative Writing: 150 human-written vs. 150 LLM-generated narratives.
   • Problem-Solving: 573 unconventional solutions vs. LLM-generated conventional solutions.
   • Research Ideation: 513 accepted papers vs. 513 rejected papers from ICLR 2024, filtered by soundness and presentation scores to isolate creativity as the differentiating factor.

3. Metrics Evaluated:
   The study assessed four metrics operating at different levels of granularity:

   • Creativity Index (CI): Measures phrase-level originality by computing n-gram overlap with large-scale web corpora (using the DJ SEARCH/Infini-Gram algorithm).
   • Perplexity (PPL): Measures token-level statistical unexpectedness based on language model prediction probabilities.
   • Syntactic Templates: Evaluates structural diversity by detecting reliance on common part-of-speech (POS) patterns (e.g., CR-POS, template rate, templates-per-token).
   • LLM-as-a-Judge: Uses LLMs (specifically Claude 3.7 Sonnet in the best configuration) with chain-of-thought (CoT) reasoning and rubric-based prompting to provide holistic, concept-level assessments.

Key Results

The analysis revealed significant inconsistencies both across domains and between metrics:

• Creativity Index (CI):

  • Domain Sensitivity: CI effectively distinguished creative from uncreative text in Creative Writing but failed to differentiate in Problem-Solving and Research Ideation.
  • Limitations: CI primarily measures lexical diversity rather than conceptual novelty. Its scores are highly sensitive to implementation choices, specifically the range of n-gram lengths ($L$) and the selection of reference corpora. Small changes in $L$ (e.g., 5–7 vs. 5–11) or the corpus can drastically alter scores, sometimes rendering the distinction between human and LLM outputs negligible.

• Perplexity (PPL):

  • Inconsistency: PPL showed overlapping score distributions between creative and uncreative sets across all domains. In some cases (e.g., Creative Writing), LLM-generated content had higher perplexity, while in others (e.g., Research Ideation), rejected papers had lower perplexity.
  • Conclusion: PPL reflects fluency and model confidence rather than novelty or creativity.

• Syntactic Templates:

  • Effectiveness: This metric struggled in domains where novel ideas are expressed through formulaic language (e.g., Research Ideation and Problem-Solving).
  • Observation: In Creative Writing, LLMs were found to rely heavily on repetitive syntactic structures (e.g., specific 8-gram templates for plot twists), whereas human writing exhibited greater structural variety. However, the metric failed to capture conceptual creativity in other domains.

• LLM-as-a-Judge:

  • Performance: While showing moderate performance in isolation, LLM judges exhibited significant instability and bias.
  • Biases: Models displayed a tendency toward one-sided predictions (e.g., classifying most solutions as "conventional" or plots as "non-original").
  • Instability: Evaluations were highly sensitive to minor prompt variations (e.g., changing "conventional" to "unconventional" in the prompt flipped model preferences). Repeated evaluations of the same instance by the same judge showed only ~40% consistency.
  • Correlation: There was a weak correlation (Pearson $r \approx 0.16$) between LLM scores and human expert scores in Research Ideation.

• Exploratory Semantic Novelty Score (SNS):

  • An experimental pipeline using semantic embeddings to measure concept-level distance failed to consistently distinguish between creative and uncreative outputs, highlighting the difficulty of capturing idea-level novelty automatically.

Key Contributions

1. Critical Analysis of Existing Metrics: The paper systematically demonstrates that current automated metrics (CI, PPL, Syntactic Templates, LLM-as-a-Judge) lack generalizability and consistency across creative domains.
2. Identification of Specific Limitations:
   • CI measures lexical rarity, not conceptual novelty.
   • PPL measures fluency, not creativity.
   • Syntactic templates miss semantic innovation in formulaic contexts.
   • LLM-as-a-Judge is prone to bias, prompt sensitivity, and low consistency.
3. Diagnostic Framework: The study provides a diagnostic groundwork for the field, clarifying where and how current metrics fail—specifically their tendency to emphasize stylistic novelty while neglecting conceptual novelty.
4. Resource Release: The authors released the curated datasets and evaluation code to facilitate future research.

Significance and Claims

The paper concludes that the current landscape of creativity evaluation is fragmented and unreliable. The findings underscore the urgent need for more robust, generalizable evaluation frameworks that better align with human judgments of creativity.

The authors modestly claim that their work does not propose a new, perfect metric but rather provides essential diagnostic groundwork. By revealing the specific gaps in existing methods, the study aims to:

• Guide practitioners to interpret existing metrics with appropriate caution.
• Inform researchers about the specific conceptual gaps that must be addressed in future work.
• Highlight the necessity of moving beyond surface-level patterns (lexical or syntactic) toward methods that can capture idea-level novelty, potentially leveraging future advances like Large Concept Models (LCM) and continuous chain-of-thought architectures.

Ultimately, the paper argues that advancing creativity assessment is essential for closing the gap between human and machine creative performance.