Technical Summary: IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs

Problem Statement

Large Language Models (LLMs) are increasingly deployed for creative tasks such as story-writing, product design, and scientific discovery. However, there is no consensus regarding their creative capabilities. While some studies suggest LLMs match or exceed human performance in generating novel and useful ideas, others highlight structural limitations, specifically fixation (constraining ideation to familiar patterns) and output homogenization (convergent responses across different models and contexts).

Existing evaluation approaches suffer from two main limitations:

1. Narrow, decontextualized tasks: Standard psychological tests like the Alternative Uses Task (AUT) or Torrance Tests (TTCT) often lack goal-oriented generation contexts, capturing diversity but failing to assess contextual utility or originality deeply.
2. Confounded broader settings: Broader evaluation settings often mix task formulation, prompting strategies, and evaluation design, making it difficult to isolate the specific effects of structured guidance on idea generation.

Furthermore, the role of structured prompting strategies—inspired by established human creativity and defixation methods (e.g., TRIZ, Design Thinking, C-K theory)—in shaping LLM idea generation remains underexplored. It is unclear whether these methods effectively mitigate fixation in LLMs or if LLMs rely merely on semantic manipulation and data patterns rather than genuine creative reasoning.

Methodology: The IDEAFix Framework

The authors introduce IDEAFix, a controlled, extensible evaluation framework designed to analyze divergent thinking in open-ended idea generation tasks. The framework follows a four-step pipeline:

1. Brief Construction

A corpus of 81 design briefs was constructed, covering four categories: Products (26), Services (15), Strategies (27), and Procedures (14). These include standard design tasks, complex scenarios (e.g., ethical dilemmas), and foundational creativity tasks from social science literature.

2. Attribute Expansion

To investigate the effect of semantic variation, each brief was expanded into 567 experimental conditions. This was achieved by manipulating the typicality of adjective attributes associated with the brief into three levels:

• Traditional: Frequently associated with the domain.
• Atypical: Rarely associated with the domain.
• Surprising: Unexpected associations.
  Attributes were generated with both positive and negative valence, guided by expert judgment to ensure syntactic consistency and contextual relevance.

3. Structured Prompt Design

The framework utilizes 25 structured prompts divided into:

• Control Prompts (2): Simple semantic variations serving as baselines.
• Method-Specific Prompts (21): Grounded in six established creativity/design methods: Brainstorming (3 variants), C-K Theory (4), TRIZ (4), Design Thinking (4), Prospective Scenarios (3), and SCAMPER (1).
• AI-Specific Strategies (4): Novel prompts designed specifically for LLMs, focusing on category negation and forcing models to avoid their simplest exploratory paths (e.g., "think outside the box," "wild," "unconventional").

All prompts share a standardized structural template to ensure comparability, including method descriptions, step-by-step reasoning instructions, the expanded brief, and shared output formatting.

4. Metrics for Divergent Thinking

Generated ideas were evaluated using six automated metrics based on semantic distances in embedding space (cosine similarity):

• Fluency: Number of ideas generated.
• Diversity: Average pairwise distance between ideas within a single output.
• Novelty: Distance of an idea from a set of "common" ideas (generated by control prompts).
• Rarity: Distance of an idea from the ensemble of all ideas generated by a model across all strategies.
• Overlap: Extent of keyword overlap between the brief and the output.
• Perplexity: Measure of sequence predictability (coherence).

Experimental Setup

The study evaluated five LLMs: Meta-Llama-3.1-70B, Qwen3-30B, Gemini-2.5-Flash, GPT-4o, and Grok-4.1-fast-reasoning. Models were run with temperatures optimized for creative outputs (0.7–1.0) and generated three independent iterations per prompt to account for stochastic variability. The dataset comprises 14,350 input prompts.

Key Results

Model Performance Variability

Performance varied significantly across models. Grok and Llama generally produced outputs with higher novelty, rarity, and diversity compared to GPT-4o and Gemini. However, all models exhibited a tendency toward output homogenization.

Impact of Briefs and Attributes

• Brief Type: Product briefs elicited the highest novelty and rarity but lower fluency. Procedure briefs generated broader, more varied outputs, while service briefs consistently produced the most conventional responses.
• Attributes: Surprising and negative keywords consistently increased novelty and rarity, often at the expense of diversity. This suggests models can be steered toward less typical associations through specific semantic framing.

Effectiveness of Prompting Strategies

• General Improvement: All method-inspired prompts outperformed control prompts in novelty and rarity, with statistically significant differences.
• Simplicity vs. Complexity: Counter-intuitively, simpler prompts (specifically the AI-specific strategies and certain Brainstorming variants) emphasizing explicit divergence cues (e.g., "wild," "unconventional") outperformed complex, method-heavy prompts (like full C-K or SCAMPER implementations) in generating top-tier novel ideas.
• Method Limitations: Complex methods like TRIZ and C-K did not necessarily maximize novelty compared to simpler divergence instructions, suggesting that LLMs respond more strongly to explicit divergence cues than to the structural complexity of human creativity frameworks.

Homogenization and Fixation

Despite improvements in individual idea novelty, the study observed persistent output homogenization across models. Solutions remained bound within a shared semantic space heavily oriented toward digital and technological innovation. This confirms the "LLM hivemind" effect, where models converge on similar outputs even when guided by defixation strategies.

Significance and Contributions

The paper claims the following contributions:

1. IDEAFix Framework: A controlled, extensible framework for studying divergent thinking in LLMs that isolates the effects of task formulation, attribute selection, and prompting strategies.
2. Structured Dataset: A dataset of 14,350 prompts combining diverse briefs, attribute expansions, and structured prompting protocols inspired by human creativity methods.
3. Systematic Analysis: Evidence that while specific prompting strategies (particularly simple, explicit divergence instructions) can boost the originality of solutions, they do not fully eliminate the inherent fixation and homogenization limits of LLMs.

The authors conclude that IDEAFix provides a necessary tool for understanding the mechanisms underlying LLM creativity, highlighting that while LLMs can be guided to produce more novel variations of existing knowledge, they struggle to generate truly transformative ideas that break beyond learned constraints. The framework allows for controlled experiments to study these limits without the susceptibility to data contamination found in static benchmarks.