Are LLMs becoming similarly creative? Evidence from three years of models

Technical Summary: "Are LLMs becoming similarly creative? Evidence from three years of models"

Problem Statement
While numerous benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, there is a lack of longitudinal data regarding LLM performance on open-ended tasks where creativity, originality, and diversity are paramount. Existing research suggests that while LLMs may appear individually creative, they often exhibit collective homogeneity, producing outputs that are similar to one another across different models and time. However, it remains unclear whether this homogeneity is a temporary artifact of developing technology or an inevitable feature of statistical language models. Current benchmarks (e.g., MMLU, HELM) focus on explicit criteria with verifiable answers and fail to capture trends in the diversity or originality of creative outputs over time.

Methodology
The authors conducted a preliminary temporal analysis of LLM output diversity using a four-step process:

1. Prompt Selection: Two complementary sets of open-ended prompts were selected to elicit creative behavior:
   • Alternate Uses Task (AUT): A standardized psychometric assessment of divergent thinking where models generate unconventional uses for common objects.
   • Infinity-Chat100: A collection of 100 real-world open-ended user queries spanning six categories, including creative content generation, brainstorming, and ideation.
2. Data Collection: Prompts were run against 68 models released between March 2023 and July 2026, representing 12 major model providers (33 closed-weight and 34 open-weight). All generations used a temperature and top-p of 1.0 to retain stochastic sampling.
3. Semantic Embedding: Responses were embedded using the all-MiniLM-L6-v2 sentence-transformer model. For AUT, all uses for a single object were embedded as a single vector; for Infinity-Chat, each response was embedded individually.
4. Regression Analysis: The study computed cosine distances between embeddings of cross-family model pairs (models from different providers) to measure semantic divergence. These pairs were grouped into nine temporal bins based on release dates. An Ordinary Least Squares (OLS) regression was performed on the mean cosine distance against the bin index. To mitigate bias from unequal family representation, the authors performed 1,000 iterations of family-balanced resampling to generate a distribution of slope estimates.

Key Contributions

• Longitudinal Framework: This paper provides the first framework for tracking the evolution of LLM creative outputs over time, moving beyond static snapshots of model behavior.
• Dual-Task Analysis: By combining a controlled psychometric task (AUT) with real-world user queries (Infinity-Chat100), the study assesses creativity across both structured and unstructured contexts.
• Quantitative Evidence of Convergence: The study offers empirical evidence that LLM outputs are becoming increasingly similar over time, challenging the assumption that newer models inherently offer greater creative diversity.

Results
The analysis reveals a statistically significant decrease in model output diversity over the three-year observation period:

• Trend: Both the AUT and Infinity-Chat datasets show a consistent negative slope in cross-family cosine distance over time, indicating increasing semantic similarity (homogeneity).
• Magnitude: The decline is most pronounced in the Alternate Uses Task, where mean cross-family distance dropped from approximately 0.50 in the earliest release bin to below 0.40 in the most recent. The Infinity-Chat dataset showed a gentler but consistent decline from ~0.34 to ~0.32.
• Robustness: All 1,000 resampling iterations produced negative slopes for both datasets, confirming that the trend is robust across different combinations of models from each family.

Significance and Claims
The paper posits that LLMs are becoming less creative in tasks requiring open-ended responses, suggesting a convergence in creative substance across models. The authors argue that if this trend persists, "LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work."

The study frames these findings as a preliminary analysis that demands careful consideration of the role of LLMs in the human creative process. It highlights a potential "AI creativity paradox": while models may individually appear creative, their aggregate outputs are becoming less diverse, potentially bounding the range of ideas users encounter and negatively impacting downstream human creativity. The authors explicitly state that their work is preliminary and call for future research to examine the underlying mechanisms (such as shared training data or distillation) and the specific factors driving this convergence.