💬 NLP

CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity

This paper introduces CreativityPrism, a scalable, cross-domain evaluation framework that assesses large language models across eight tasks in divergent thinking, creative writing, and logical reasoning using three dimensions (quality, novelty, and diversity), revealing that while frontier models excel in writing and reasoning, they show no significant advantage in divergent thinking and that performance rarely generalizes across different creative dimensions.

Original authors: Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, magical library of text generators (Large Language Models, or LLMs). Everyone is asking: "Are these machines actually creative?"

The problem is that "creativity" is a slippery concept. Sometimes it means writing a funny story, sometimes it means solving a math problem in a weird way, and sometimes it means thinking of a new use for a paperclip. Before this paper, researchers were trying to measure creativity with different rulers for different jobs, or they were asking humans to grade every single answer, which is slow and expensive.

Enter "CreativityPrism."

Think of CreativityPrism as a new, high-tech prism that you shine a beam of light (the AI's output) through. Instead of just seeing a white blob, the prism splits the light into three distinct colors, revealing the true nature of the machine's creativity.

The Three Colors of Creativity

The authors argue that to truly understand if a machine is creative, you have to measure it in three specific ways:

Quality (The "Does it Work?" Color): This is the foundation. If a machine writes a story that makes no sense or solves a math problem incorrectly, it doesn't matter how "new" the idea is. It's just noise. Quality measures if the output actually fulfills the task.
Novelty (The "How Weird Is It?" Color): This measures how far the answer strays from the ordinary. Did the machine come up with a solution no human has ever seen before? Or did it just copy what's in its training data?
Diversity (The "How Many Options?" Color): This measures the breadth. If you ask the machine for 10 ideas, are they 10 totally different ideas, or are they just 10 slightly different versions of the same idea?

The Analogy: Imagine a chef.

Quality: The food tastes good and isn't burnt.
Novelty: The chef invents a flavor combination no one has ever tried (e.g., chocolate and pickles).
Diversity: The chef can make 10 different types of desserts, not just 10 variations of chocolate cake.
CreativityPrism checks all three. A chef who makes 100 different versions of a burnt chocolate cake has high diversity but low quality and low novelty.

The Big Test: 17 Chefs in the Kitchen

The authors took 17 of the smartest AI models currently available (from companies like OpenAI, Google, Anthropic, and DeepSeek) and put them through 8 different challenges across three areas:

Divergent Thinking: Like the "Alternative Uses Test" (e.g., "What can you do with a brick other than build a wall?").
Creative Writing: Writing stories or poems based on prompts.
Logical Reasoning: Solving math problems or writing code under strict, weird constraints.

What They Found (The Plot Twist)

1. The "Big vs. Small" Gap
The massive, expensive AI models (the "Frontier" models) are generally better at Creative Writing and Logical Reasoning. They are like master chefs who can follow complex recipes and write beautiful menus. However, when it comes to Divergent Thinking (coming up with wild, unrelated ideas), the gap between the super-expensive models and the smaller, open-source models almost disappears. The big models aren't necessarily better at "thinking outside the box" than the smaller ones.

2. The "Specialist" Problem
Here is the most surprising finding: Being good at one type of creativity doesn't mean you are good at another.

A model might be amazing at writing a novel (High Quality) but terrible at coming up with weird new uses for a spoon (Low Novelty).
A model might generate 1,000 different answers (High Diversity) but none of them are actually new or useful (Low Novelty).

The authors found that "Novelty" is the most chaotic dimension. Just because a model is good at being "new" in a math problem doesn't mean it will be "new" in a story. They are totally different skills.

3. The "Judge" Solution
Since humans are too slow to grade all these tests, the authors used a clever trick: they used other AIs to grade the answers. But they didn't just trust them blindly. They had human experts grade a small sample first to make sure the "AI Judges" were actually agreeing with humans. It's like having a head chef taste a few dishes to make sure the sous-chefs are grading the food correctly before they grade the whole kitchen.

The Bottom Line

CreativityPrism is a new tool that stops us from saying "This AI is creative" as a single, vague statement. Instead, it tells us: "This AI is great at writing good stories, okay at solving math creatively, but not very good at coming up with wild, unrelated ideas."

It proves that creativity isn't one single superpower; it's a collection of different skills that don't always travel together. To build truly creative machines, we need to train them specifically for these different "colors" of creativity, not just hope they get it all right by accident.

Technical Summary: CreativityPrism

Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) in generating creative text, there is currently no unified, cross-domain, and scalable framework to evaluate LLM creativity. Existing evaluation methods suffer from two primary limitations:

Fragmentation: Current research is scattered across specific domains (e.g., divergent thinking, creative writing, coding, math) with narrow definitions of creativity. Benchmarks often focus on single dimensions (e.g., lexical diversity or unconventional ideas) and vary in the models they test, making cross-comparison difficult.
Scalability: Creativity is inherently subjective, leading many existing benchmarks to rely heavily on human evaluation. This approach is prohibitively expensive, slow, and difficult to replicate at scale, creating a bottleneck for the rapid iteration required by modern AI research.

Methodology

The authors propose CreativityPrism, a cross-domain evaluation framework designed to consolidate creativity assessment into a scalable, automatic pipeline.

Framework Design

Task Selection: The framework integrates eight tasks across three distinct domains:
- Divergent Thinking: Alternative Uses Test (AUT), Divergent Association Task (DAT), and Torrance Tests of Creative Thinking (TTCT).
- Creative Writing: Torrance Test of Creative Writing (TTCW), Creative Short Story, and Creativity Index.
- Logical Reasoning: Creative Math and NeoCoder (coding under constraints).
Taxonomy of Dimensions: Moving beyond simple task aggregation, the authors categorize all evaluation metrics into three theoretical dimensions of creativity, grounded in the Torrance Tests of Creative Thinking (TTCT):
1. Quality: The ability to produce coherent, well-formed, and functional responses (e.g., code correctness, narrative fluency).
2. Novelty: The ability to produce outputs that deviate from common or existing solutions (e.g., uniqueness compared to training corpora or reference solutions).
3. Diversity: The breadth of the output distribution, measuring how widely a model varies its responses across generations (e.g., semantic distance between generated nouns).
Scalable Evaluation (LLM-as-a-Judge): To ensure scalability, the framework utilizes automatic evaluation. For metrics requiring subjective judgment, the authors employ an LLM-as-a-Judge paradigm. Crucially, they validate these judges using the Alternative Annotator Test (Calderon et al., 2025), ensuring that the chosen LLM judges align closely with high-quality human annotations (Fleiss Kappa $\ge$ 0.4) before being deployed.

Experimental Setup

Models Evaluated: 17 State-of-the-Art (SoTA) LLMs were benchmarked, including frontier-scale models (GPT-4.1, Claude 3, Gemini 2.0, DeepSeek-V3/R1) and locally deployable open models (Mistral, Qwen, Llama, OLMo families).
Inference: Standardized inference settings were applied (Temperature 0.75, max tokens 4096), with specific adjustments for tasks like Creative Math (Temperature 0.0) and Creativity Index (Temperature 1.0).
Scoring: Raw metrics were min-max normalized and aggregated into the three dimensions (Quality, Novelty, Diversity) and an overall creativity score.

Key Results

The evaluation of 17 LLMs yielded several significant findings:

Performance Gap: Frontier-scale models significantly outperform locally deployable open models. The best frontier models lead the best open models by approximately 0.10 (15%) across creativity dimensions.
- This gap is most pronounced in Logical Reasoning and Creative Writing.
- The gap is smaller in Divergent Thinking, suggesting this domain is less prioritized in current post-training regimes compared to reasoning and writing.
- The Quality dimension shows the largest performance disparity, likely due to the heavy emphasis on reasoning tasks in frontier model training.
Lack of Generalization: High performance in one creative dimension or domain does not guarantee success in others.
- Novelty metrics show weak or even negative correlations with other metrics. For instance, a model might generate highly diverse outputs (high diversity) that are individually unoriginal, or produce a single novel output with no variation (high novelty, low diversity).
- This fragmentation confirms that creativity is not a monolithic capability; a model's "creativity" is highly task- and dimension-dependent.
Model Rankings: Among open models, Qwen2.5-72B emerged as the top performer. Among frontier models, DeepSeek-V3 and GPT-4.1 demonstrated the highest overall creative capabilities.

Significance and Contributions

The paper claims the following contributions:

Unified Framework: It introduces CreativityPrism, the first framework to systematically combine eight tasks from three domains into a single, scalable evaluation pipeline.
Dimensional Taxonomy: It proposes a "Quality, Novelty, and Diversity" taxonomy, arguing that separating "originality" into "novelty" (depth/deviation) and "diversity" (breadth/variance) is essential for nuanced LLM evaluation.
Reliable Automation: It validates the use of LLM-as-a-Judge for creativity assessment through rigorous human alignment testing (Alternative Annotator Test), offering a scalable alternative to manual grading.
Empirical Insights: The benchmark reveals that current LLMs exhibit fragmented creative abilities, with frontier models excelling in reasoning-driven creativity but showing no significant advantage in divergent thinking. This highlights the need for cross-domain frameworks to avoid misleading conclusions about model capabilities.

Limitations

The authors acknowledge several limitations:

Modality: The framework is currently limited to text; multimodal creativity is not covered due to the immaturity of automatic evaluation methods in that space.
Domain Coverage: Highly creative domains like scientific discovery and inventive design are excluded due to a lack of reliable automatic evaluation metrics.
Human Validation: While LLM judges were validated, the human annotators were primarily CS researchers rather than creativity domain experts, and sample sizes were constrained by resources.
Data Contamination: Models with later release dates may have an advantage on novelty metrics if their training data overlaps with the benchmark's reference corpora.

In conclusion, the paper argues that a cross-domain, multi-dimensional framework like CreativityPrism is essential for any meaningful assessment of LLM creativity, as single-domain or single-metric evaluations fail to capture the complexity of machine creativity.