Examining and Addressing Barriers to Diversity in LLM-Generated Ideas

1. Problem Statement

While Large Language Models (LLMs) excel at generating high-quality and novel ideas individually, they suffer from a critical deficiency at the collective level: a lack of idea diversity compared to human populations.

• The Phenomenon: Independent LLM samples frequently converge on similar themes and concepts (a phenomenon known as "mode collapse"), whereas independent human samples exhibit significantly higher diversity.
• The Consequence: This homogenization threatens the innovation ecosystem. Since idea generation is a bottleneck in the innovation process, a lack of initial diversity cannot be recovered by downstream selection or refinement. Widespread reliance on LLMs risks creating a "tragedy of the commons" where individual efficiency gains lead to collective stagnation in solution space exploration.
• The Gap: Existing literature identifies prompting strategies (e.g., Chain-of-Thought, Personas) that improve diversity but lacks a theoretical framework explaining why LLMs lack diversity and how these strategies mechanistically address the root causes. Furthermore, prior comparisons often lack "apples-to-apples" experimental controls.

2. Theoretical Framework

The authors propose a framework grounded in cognitive psychology, identifying two distinct mechanisms that limit LLM diversity:

1. Fixation (Individual Level):
   • Human Context: Humans experience fixation where early ideas or examples constrain subsequent ideation (e.g., part-list cueing, unconscious plagiarism).
   • LLM Context: The authors argue LLMs exhibit analogous fixation. Due to the autoregressive nature of generation (predicting the next token based on prior context) and RLHF alignment (which favors conventional responses), early outputs anchor subsequent generations, causing the model to explore a narrow trajectory within a single session.
2. Knowledge Partitioning (Collective Level):
   • Human Context: Human knowledge is partitioned across billions of distinct minds. Each person occupies a unique "region" of the knowledge space, leading to diverse starting points and divergent evolutionary paths for ideas.
   • LLM Context: LLMs aggregate all training data into a single, unified probability distribution. They lack the inherent partitioning of human cognition. Consequently, independent LLM sessions sample from the same central tendency rather than distinct regions, leading to low collective diversity.

3. Methodology

The authors conducted four studies using a rigorous, controlled experimental design to isolate these mechanisms.

• Task: Generating 10 fitness product ideas.
• Participants:
  • Humans: Recruited via Prolific ($N \approx 99$ per study).
  • LLMs: Simulated using GPT-4o ($N \approx 99$ independent API calls per study).
• Experimental Controls:
  • Matched Conditions: Humans and LLMs were given identical constraints (e.g., generating one idea at a time, viewing previous ideas).
  • Novel Diversity Metric: Instead of relying solely on embedding similarity (which can be skewed by LLMs' uniform phrasing), the authors developed a hierarchical content categorization system. Ideas were classified into 28 unique categories across three dimensions: Industry Context, Psychological Need, and Product Form. Diversity was measured by:
    1. Total unique categories ($T_{cat}$).
    2. Total unique category combinations ($T_{comb}$).
    3. Average pairwise distance ($d$).
• Interventions Tested:
  • Seeding: Providing LLMs with human-generated first ideas.
  • Personas: Assigning LLMs "Creative Entrepreneurs" (e.g., Steve Jobs) vs. "Ordinary Personas" (randomly sampled diverse backgrounds).
  • Chain-of-Thought (CoT): A multi-step prompting strategy requiring the model to generate titles, revise them for distinctness, and then expand them.

4. Key Results

Study 1: Establishing the Gap and Mechanisms

• Diversity Gap: Humans generated significantly more diverse ideas than default LLMs (206 unique combinations vs. 88).
• Fixation: Both humans and LLMs exhibited similar levels of within-individual fixation (diversity accumulation rates were statistically identical). Thus, fixation alone does not explain the collective gap.
• Knowledge Partitioning: The gap was driven by between-individual differences. LLMs' first ideas were significantly less diverse than humans', indicating they sample from a centralized distribution rather than distinct knowledge regions.

Study 2: The Limits of Seeding

• Finding: Seeding LLMs with human first ideas did not improve subsequent diversity.
• Implication: The diversity gap is not just about the starting point; it is about how ideas evolve. LLMs lack the unique associative structures (mental models) that allow humans to diverge from the same starting point.

Study 3: The Role of Personas

• Finding: Assigning Ordinary Personas (randomly sampled diverse backgrounds) significantly boosted LLM diversity, matching or exceeding human levels in unique combinations (210 vs. 206).
• Comparison: "Ordinary Personas" outperformed "Creative Entrepreneurs" (e.g., Elon Musk). The authors argue creative archetypes are semantically clustered in the model's training data, whereas ordinary personas act as diverse sampling cues that push the model into distinct semantic regions.
• Trade-off: While personas improved collective diversity (knowledge partitioning), Ordinary Personas increased within-individual fixation (the model got "stuck" in the persona's specific worldview).

Study 4: Chain-of-Thought (CoT) and Combination

• CoT Effect on LLMs: CoT prompting significantly reduced fixation in LLMs (increasing the slope of diversity accumulation) by forcing structured revision and breaking the autoregressive anchor.
• CoT Effect on Humans: CoT did not improve human diversity and slightly reduced it, likely due to "ironic process" theory (trying to force difference paradoxically reinforces the original thought).
• The Optimal Solution: Combining Ordinary Personas (to recover knowledge partitioning) with CoT (to reduce fixation) produced the highest diversity.
  • Result: The combined approach yielded 248 unique combinations, surpassing human performance by 26%.

5. Key Contributions

1. Theoretical Framework: First to propose a cognitive psychology-based framework explaining LLM diversity limitations via Fixation and Lack of Knowledge Partitioning.
2. Methodological Innovation: Introduced a content-based hierarchical categorization metric that avoids the biases of embedding-based similarity measures when comparing humans and LLMs.
3. Empirical Discovery: Demonstrated that Ordinary Personas are superior to "Creative" archetypes for diversity because they provide better sampling cues for the model's knowledge space.
4. Human-AI Divergence: Showed that CoT reduces fixation in LLMs but not in humans, highlighting a fundamental difference in how instruction-following impacts cognitive flexibility in machines vs. people.
5. Surpassing Human Performance: Provided the first evidence that LLMs, when guided by theory-driven interventions, can exceed human collective diversity.

6. Significance and Implications

• For Innovation Strategy: Organizations should not rely on default prompts. To maximize innovation, they should deploy synthetic personas representing diverse, non-expert backgrounds to simulate human knowledge partitioning, combined with CoT prompting to prevent the model from getting stuck.
• Mitigating the "Tragedy of the Commons": By using customized, diverse personas, firms can avoid the convergence of ideas that occurs when everyone uses the same generic LLM prompts.
• Future of AI Collaboration: The findings suggest that the most effective human-AI collaboration leverages AI's ability to follow complex structural instructions (CoT) while using AI to simulate the cognitive diversity of a human crowd (Personas), effectively unlocking a broader solution space than either humans or raw LLMs can achieve alone.