Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

1. Problem Statement

While Large Language Models (LLMs) are increasingly used as substitutes for human participants in cognitive science experiments, their ability to replicate human behavioral variability remains unverified. Previous research suggests LLMs lack the linguistic diversity and individual differences characteristic of human cognition, often producing rigid, high-frequency outputs.

This study addresses a critical gap: Can LLMs simulate the inter-participant variability observed in human phonemic fluency tasks? Unlike semantic fluency (which relies on meaning), phonemic fluency (generating words starting with a specific letter, e.g., "F") requires effortful, form-based retrieval, making it a stringent test of whether LLMs can mimic the stochastic and diverse nature of human memory search.

2. Methodology

Data Sources

• Human Data: 106 native English speakers recruited via Amazon Mechanical Turk. Participants generated words starting with "F" within one minute. Data was transcribed and cleaned (removing repetitions, proper nouns, etc.), yielding an average of 16.89 words per participant.
• LLM Simulation: The study evaluated 34 distinct models across 45 configurations from major providers (OpenAI, Anthropic, Google, xAI) and open-source models (Meta, Alibaba, DeepSeek, etc.).
• Prompting Strategy: Models were prompted to role-play specific human participants. The prompt included the participant's age, education level, and actual number of correct responses (to constrain output volume) and instructed the model to generate words starting with "F" for one minute.

Experimental Design

1. Individual Model Evaluation: Each model configuration was run 106 times (once per human participant).
2. Ensemble Simulation: To test if combining models could recover diversity, the authors created 1,000 ensemble samples where each simulated participant's response was drawn from a randomly selected model configuration.
3. Analysis Metrics:
   • Response Count: Mean Absolute Error (MAE) against human averages.
   • Lexical Diversity: Type-to-Token Ratio (TTR) and Idiosyncratic Type-to-Total Type Ratio (ITTTR).
   • Distribution: Zipf's Law analysis (scaling exponent $\alpha$) to measure frequency distribution steepness.
   • Linguistic Predictors: Stepwise regression to identify factors (word frequency, age of acquisition, orthographic neighborhood) driving word choice.
   • Network Structure: Construction of word co-occurrence networks using Triangulated Maximally Filtered Graphs (TMFG) to analyze clustering and global efficiency.

3. Key Results

A. Response Volume and Adherence

• Most models (33/45) adhered to the instructed response counts (MAE $\le$ 1.69).
• However, newer models (e.g., GPT-4 Turbo, o3) and some "Thinking" modes significantly overproduced words, failing to respect the one-minute constraint.

B. Lexical Diversity (The Core Finding)

• Human Baseline: Humans produced 476 unique word types (TTR = 0.27) with high idiosyncrasy (42% of types were unique to one participant).
• LLM Performance: No single LLM approached human-level variability.
  • The best-performing model, Claude 3.7 Sonnet, produced only 226 unique types (TTR = 0.13).
  • Thinking/Reasoning Modes: Contrary to intuition, enabling "Thinking" modes (e.g., in Claude 3.7 Sonnet) reduced diversity compared to non-thinking modes.
  • Ensemble Failure: Combining outputs from diverse models did not recover human-level diversity. The ensemble mean TTR was 0.10, far below the human 0.27.

C. Distribution and Linguistic Predictors

• Zipf's Law: Human data followed a standard power law ($\alpha = 0.89$). LLMs exhibited steeper declines ($\alpha$ ranging from 1.19 to 1.53), indicating an over-reliance on high-frequency, prototypical words and a lack of a "long tail" of rare words.
• Predictors:
  • Humans: Word choice was driven by word frequency, orthographic neighborhood size (words differing by one letter), and age of acquisition.
  • LLMs: While sensitive to word frequency and age of acquisition, LLMs (specifically Claude 3.7 Sonnet) relied heavily on word length rather than orthographic neighborhood, suggesting a different retrieval mechanism.

D. Network Structure

• Clustering: Human networks showed higher local clustering (tighter semantic/associative groups) but longer shortest path lengths (weaker global integration).
• LLM Networks: Claude 3.7 Sonnet networks had lower clustering but shorter path lengths, indicating a more evenly connected, globally efficient structure that lacks the "clumpy" organization of human associative memory.

E. The Cause of Failure: Vocabulary Overlap

• The ensemble approach failed because models share a highly overlapping vocabulary.
• Pairwise overlap coefficients were extremely high (mean = 0.74; within Anthropic models, mean = 0.90).
• This suggests that despite architectural differences, models are trained on similar internet-scale data, leading to a shared "core" of high-frequency words and a systematic suppression of rare, idiosyncratic responses.

4. Key Contributions

1. Comprehensive Benchmarking: The study is the first to evaluate 34 models and 45 configurations against a single human dataset, moving beyond single-model case studies.
2. Identification of "Thinking" Trade-offs: It reveals that enabling reasoning/thinking modes in LLMs can paradoxically reduce behavioral variability, making them less human-like in tasks requiring diverse output.
3. Ensemble Limitations: It demonstrates that simple ensemble sampling cannot overcome the fundamental lack of diversity in LLMs due to shared training data biases.
4. Structural Divergence: It provides evidence that even the most "human-like" LLM (Claude 3.7 Sonnet) organizes knowledge differently (globally efficient vs. locally clustered) compared to human memory.

5. Significance and Implications

• Limitations in Cognitive Simulation: LLMs are effective at simulating average human performance (central tendencies) but fail to capture individual differences and the full spectrum of behavioral variability.
• Research Ethics: Using LLMs as direct substitutes for human participants in studies focused on individual differences, diversity, or rare behaviors is risky. It may systematically reinforce dominant linguistic patterns while marginalizing the diversity inherent in real human populations.
• Future Directions: The authors suggest treating LLMs as a baseline system for central tendencies rather than a replacement for human participants. Future work should focus on targeted persona modeling (incorporating specific reading exposures or cognitive profiles) rather than generic prompting to better simulate human variability.

Conclusion: While LLMs can generate fluent and plausible responses, they fundamentally lack the stochastic, diverse, and structurally distinct retrieval dynamics of human cognition. They simulate the "wisdom of the crowd" but fail to simulate the "variability of the individual."