💬 NLP

How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning

This study demonstrates that multimodal large language models can effectively evaluate visual creativity in a zero-shot setting with substantial alignment to human ratings, while their step-by-step reasoning provides interpretable insights into the evaluation process despite not improving scoring accuracy.

Original authors: William Orwig, Roger E. Beaty

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: William Orwig, Roger E. Beaty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart art critic who has never taken a class on art, never seen a human grade a drawing, and has never been told what "creativity" looks like. You just hand them a picture and ask, "On a scale of 1 to 5, how creative is this?"

That is essentially what this paper does. The researchers asked six different advanced AI models (think of them as different "digital judges") to do exactly that. They wanted to see if these AIs could guess how humans would rate the creativity of pictures, without any special training or practice.

Here is the story of what they found, broken down into simple parts:

1. The "Zero-Shot" Magic Trick

Usually, to teach an AI to grade art, you have to show it thousands of examples of "good" and "bad" drawings along with human scores. It's like teaching a student by showing them a stack of graded exams.

But these researchers tried something different: Zero-Shot. They gave the AI just the picture and a simple instruction: "Rate this 1 to 5." No examples, no training. It was like asking a stranger on the street to grade a painting they've never seen before.

The Result: Surprisingly, the AI judges were pretty good at it. Their ratings matched human ratings quite well.

For AI-generated images (polished pictures made by computers), the AI judges and humans agreed very strongly.
For hand-drawn sketches (rough, messy drawings by people), the agreement was still there, but a bit weaker.

2. The "Polished vs. Rough" Bias

The researchers noticed the AI judges had a funny personality quirk, like a critic who loves high-definition photos but hates pencil sketches.

The "Polished" Bias: When looking at slick, computer-generated images, the AI judges were too generous. They gave higher scores than humans did.
The "Rough" Bias: When looking at simple, hand-drawn sketches, the AI judges were too harsh. They gave lower scores than humans did.

It seems the AI models secretly love "elaborate" and "complex" images. If a drawing has a lot of lines and detail, the AI thinks it's creative, even if the human judge thinks it's just "busy." The AI models seemed to confuse "detailed" with "creative."

3. The "Thought Process" (The Chain of Thought)

Some of these AI models have a special feature: before they give a final score, they write out their "thought process." It's like a student showing their work on a math test.

The researchers peeked inside these thought processes to see how the AI decided on a score. They found the AI follows a consistent four-step recipe:

Perception: "I see a cat and a tree." (Describing what is there).
Originality: "This is weird and new!" (Judging if it's unique).
Quality: "The lines are smooth and pretty." (Judging how well it's drawn).
Justification: "So, I give it a 4." (Picking the number).

The Big Surprise: The researchers thought that if the AI took the time to "think" and write out these steps, it would get better at matching human scores. It didn't. The "thinking" didn't make the scores more accurate. However, it did give the researchers a window into why the AI gave the score it did. It showed that the AI was paying attention to "quality" (how pretty it is) rather than just "originality" (how weird it is), which explains why it liked the polished AI images so much.

4. Does the AI Need to Know What It's Looking At?

The researchers wondered: "If the AI gets confused and thinks a drawing of a dog is a cat, does its creativity score go down?"

The Answer: Not really.
Even when the AI misidentified what the drawing was, or when the drawing was so abstract that humans couldn't agree on what it was, the AI's creativity score still matched human scores pretty well. This suggests the AI isn't just checking a checklist of "Is this a dog? Yes/No." It's looking at the vibe, the composition, and the weirdness of the image, even if it doesn't know exactly what the object is.

The Takeaway

This paper shows that we can use these big, general AI models as "creativity judges" without needing to train them first. They are surprisingly good at it, but they have a bias: they love complex, detailed images and dislike simple, rough sketches.

By looking at their "thought processes," we can see exactly where they are getting it right and where they are getting it wrong. It's like having a critic who can explain their reasoning, even if their taste is a little different from ours. The researchers even built a free app so anyone can try this "AI judge" on their own pictures.

Technical Summary: How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning

Problem Statement
Evaluating the originality of visual images remains a significant bottleneck in creativity research. While automated scoring using Large Language Models (LLMs) has demonstrated effectiveness in verbal domains (e.g., the Alternate Uses Task), its application to visual creativity is less established. Existing approaches in the figural domain often rely on task-specific training or fine-tuning on large datasets of human-rated examples, limiting their generalizability. Furthermore, there is a lack of understanding regarding the latent processes by which AI models arrive at their ratings. The central question addressed is whether general-purpose, multimodal LLMs can serve as "zero-shot" judges of visual creativity—without fine-tuning, exemplars, or additional training—and whether their explicit reasoning chains offer an interpretable window into their evaluation logic.

Methodology
The research employed a two-study design to evaluate six multimodal LLMs (Gemini 3 Flash, Gemma 4 31B IT, GPT-5.4 Mini, GLM-5v Turbo, Kimi K2.5, and Qwen 3.6 Plus) across two distinct visual creativity datasets:

AI-Generated Images: 992 images created by DALL-E 3 from human-written word prompts, rated by human experts on a 1–5 scale.
Hand-Drawn Sketches: 1,500 sketches from the AuDrA platform, where participants extended incomplete starting shapes. These were rated using a Judge Response Theory (JRT) model to derive continuous creativity scores.

Study 1: Zero-Shot Alignment
All six models were tested in a zero-shot setting, receiving only an image and a text prompt instructing them to rate creativity on a 1–5 scale. The study measured:

Alignment: Pearson correlations between model ratings and human criterion scores.
Complexity Control: Partial correlations controlling for "edge density" (a proxy for visual complexity) to ensure models were not merely rewarding elaboration.
Bias Analysis: Examination of rating distributions to identify leniency or severity biases relative to human raters.

Study 2: Reasoning Chain Analysis
Three reasoning-capable models (GLM-5v Turbo, Kimi K2.5, Qwen 3.6 Plus) were re-evaluated with reasoning enabled. The study analyzed the explicit "chains of thought" produced before the final rating:

Chain Validation: A separate text-only LLM attempted to predict the final rating based solely on the reasoning chain (with numeric digits removed) to verify if the chain contained the evaluative signal.
Process Coding: Reasoning chains were decomposed into sentences and classified into four evaluative dimensions: Perception (describing content), Originality (judging novelty), Quality (judging aesthetics/composition), and Justification (anchoring the rating).
Perceptual Accuracy: Models' ability to identify the depicted object in sketches was compared against human consensus labels to determine if object recognition errors impacted creativity alignment.

Key Results

Zero-Shot Feasibility: All six models showed substantial alignment with human ratings on both datasets without fine-tuning. Correlations ranged from $r = .57$ to $.68$ for AI-generated images and $r = .29$ to $.68$ for hand-drawn sketches. Gemini 3 Flash demonstrated the strongest alignment across both datasets ( $r = .68$ ).
Beyond Visual Complexity: Partial correlations controlling for edge density remained robust, indicating that models detect higher-level creative features rather than simply rewarding visual elaboration.
Systematic Biases: Models exhibited a "leniency bias" toward polished AI-generated images (rating them higher than humans) and a "severity bias" toward sparse hand-drawn sketches (rating them lower), suggesting a preference for outputs resembling their own training distribution.
Reasoning Impact: Engaging reasoning chains did not improve alignment with human ratings compared to the non-reasoning baseline; in some cases (e.g., Qwen 3.6 Plus), reasoning slightly reduced alignment.
Interpretability: Reasoning chains were highly interpretable and contained the evaluative signal driving the ratings (a text-only model could predict the rating from the chain with $r = .82$ ). The chains followed a consistent structure: Perception $\rightarrow$ Originality/Quality assessment $\rightarrow$ Justification.
Divergence Drivers: The proportion of "Originality" sentences in a chain correlated with lower model ratings relative to humans, while "Quality" sentences correlated with higher ratings.
Perceptual Independence: Correct object identification was not a prerequisite for high alignment. Misidentified drawings still showed strong model-human correlation ( $r = .43$ vs. $r = .49$ for correct matches). Notably, the highest alignment occurred for "No-Consensus Drawings" (images humans could not categorize), suggesting models and humans converge on structural and compositional features when object identity is ambiguous.

Significance and Claims
The paper claims to provide the first evidence that general-purpose multimodal LLMs can match human judgments of visual creativity in a zero-shot setting, extending automated assessment beyond verbal tasks to the figural domain without task-specific training.

Key contributions include:

Validation of Zero-Shot Scoring: Demonstrating that prompt-based evaluation is sufficient for visual creativity assessment across diverse stimulus types.
Interpretability of Evaluation: Revealing that LLMs evaluate creativity through a consistent, four-stage reasoning process (perception, originality, quality, justification), offering a transparent view into AI evaluation mechanisms.
Decoupling of Recognition and Evaluation: Establishing that creativity scoring by LLMs is not reducible to object recognition; models can assess creative value even when they fail to identify the specific subject of a drawing.
Bias Detection: Highlighting specific biases (leniency toward AI-generated content, severity toward sketches) that must be addressed in future automated scoring pipelines.

The authors conclude that while multimodal LLMs offer a promising tool for scalable visual creativity assessment, their reasoning traces are most valuable for auditing how they evaluate, rather than for improving the raw accuracy of the ratings themselves. The authors caution against applying these zero-shot methods to unvalidated tasks and emphasize the need for careful calibration to mitigate inherent biases. An open-source web app implementing this pipeline is provided to facilitate further research.