The threat of analytic flexibility in using large language models to simulate human data

1. Problem Statement

The paper addresses a critical methodological threat in the emerging use of Large Language Models (LLMs) as "silicon samples" (synthetic datasets intended to replace human respondents in social science research). While LLMs offer promises of efficiency, cost reduction, and the ability to simulate hard-to-reach populations, their application suffers from extreme analytic flexibility.

Unlike traditional human data collection, generating silicon samples involves a vast "garden of forking paths" where researchers must make numerous defensible but arbitrary analytic choices, including:

• Model selection (e.g., GPT-4 vs. Llama vs. reasoning models).
• Hyperparameters (e.g., temperature, top-k, top-p, reasoning effort).
• Prompt engineering (e.g., item-by-item vs. scale-by-scale, narrative vs. bullet points).
• Contextual conditioning (e.g., amount and type of demographic information provided).

The author argues that without rigorous standards, this flexibility allows researchers to inadvertently (or intentionally) select configurations that produce specific results, leading to unreplicable findings and a literature grounded in arbitrary methodological decisions rather than substantive truth.

2. Methodology

The study employs a specification curve analysis (multiverse analysis) across two distinct studies to quantify how analytic choices affect the fidelity of silicon samples compared to ground-truth human data.

Study 1: Controlled Case Study

• Data Source: 85 participants from the Attitudes, Identity, and Individual Differences (AIID) dataset.
• Measures: Two social-psychological scales:
  1. Gut Feelings Scale: Racial bias (African-Americans vs. European-Americans).
  2. Belief in a Just World (BJW) Scale.
• Experimental Design: The author generated 252 distinct silicon-sample configurations by varying four key dimensions:
  1. Model: 9 different models (e.g., ChatGPT-3.5, GPT-4o, Llama 3.3, DeepSeek-R1).
  2. Hyperparameters: Temperature (0–1.5) or Reasoning Effort (High/Low).
  3. Demographics: None, Age/Gender only, or Extensive (7 variables).
  4. Prompt Strategy: All-in-one, Scale-by-scale, or Item-by-item.
• Evaluation Metrics:
  1. Ranking: Correlation between human and LLM participant rankings (ordinal preservation).
  2. Distribution: Wasserstein distance between human and LLM response distributions.
  3. Relationship: Correlation between the two scales (BJW vs. Gut Feelings).

Study 2: Re-analysis of Published Work

• Target: Re-examined Argyle et al. (2023), Study 3, a highly cited paper claiming silicon samples could replicate human association structures.
• Data Source: 500 respondents from the 2016 American National Election Studies (ANES).
• Task: Predicting 11 political/attitudinal variables based on 12 interview variables.
• Experimental Design: Generated 66 configurations varying model, temperature, prompt style (interview vs. narrative vs. bullet points), and conditioning information (full vs. reduced demographics).
• Evaluation Metric: Correlation between the matrix of human pairwise associations (Cramer's V) and the matrix of silicon-sample associations.

3. Key Results

Study 1 Findings

• Substantial Variability: Analytic choices caused massive fluctuations in performance.
  • Ranking: Correlations between human and LLM rankings ranged from -0.27 to 0.36 (very low).
  • Distributions: Wasserstein distances for LLMs were significantly higher (worse) than the distance between two random splits of human data.
  • Correlations: The estimated correlation between the two scales varied wildly from -0.40 to 0.78, despite the true human correlation being 0.26.
• Lack of Consistency: High performance on one metric did not predict high performance on another. A configuration that accurately modeled response distributions often failed to capture participant rankings or inter-scale correlations.
• No "Best" Configuration: No single model or setting consistently outperformed others across all metrics. Interestingly, configurations with no demographic information sometimes outperformed those with extensive demographics, contradicting the premise of silicon sampling.
• Stochasticity: Even with temperature set to 0 (minimizing randomness), the specification spread remained large, suggesting the variability is structural to the configuration choices, not just noise.

Study 2 Findings

• Impact on Published Conclusions: Re-running Argyle et al.'s task with different configurations yielded correlation coefficients between human and silicon association structures ranging from r = 0.23 to r = 0.84.
• Original vs. Alternative: Argyle et al.'s original configuration (GPT-3, specific prompt) yielded r = 0.72. However, other defensible configurations produced results as low as r = 0.23 (weak correspondence) or failed entirely due to refusals to generate demographic data.
• Sensitivity: The "best" performing configuration in this study (GPT-3.5, temp 1.5, narrative prompt) was different from the original study's setup, demonstrating that the original conclusion of high fidelity was contingent on specific, unexamined choices.

4. Key Contributions

1. Empirical Demonstration of Flexibility: This is the first dedicated empirical investigation showing that a small set of analytic decisions can materially alter the conclusions of silicon sample research, rendering them as flexible as traditional p-hacking in human studies.
2. Decoupling of Metrics: The paper proves that "fidelity" is not a monolithic property; a model can be good at ranking participants but poor at capturing distributions, or vice versa.
3. Re-evaluation of Existing Literature: By showing that a highly cited paper's results (Argyle et al., 2023) are highly sensitive to configuration, the study suggests that much of the existing silicon sample literature may be fragile and non-replicable.
4. Calibration-Confirmation Framework: The author proposes a machine-learning-inspired workflow (train-test splitting) for silicon samples, where researchers must use a "calibration" subset of human data to select a configuration before applying it to a "confirmation" subset.

5. Significance and Implications

• Threat to Validity: The current state of silicon sampling risks creating a literature filled with unreplicable findings driven by arbitrary configuration choices rather than true psychological signals.
• Generalizability Crisis: The study finds that configurations do not generalize across domains. A setup that works for political science (Study 2) may fail for social psychology (Study 1), and vice versa.
• Ethical Concerns: Using silicon samples to simulate vulnerable or hard-to-reach populations is particularly dangerous. If models cannot accurately simulate typical WEIRD (Western, Educated, Industrialized, Rich, Democratic) populations due to analytic flexibility, they are unlikely to accurately represent underrepresented groups, potentially creating an "illusion of representation" that silences marginalized voices.
• Call for Rigor: The author calls for:
  • Preregistration of analytic choices and evaluation criteria.
  • Specification Curves to visualize how results vary across the "multiverse" of possible configurations.
  • Calibration-Confirmation workflows requiring human benchmark data to validate specific configurations before use.
  • Critical Reconsideration: Researchers should question whether silicon samples are necessary if they require as much effort to calibrate as collecting human data, and whether alternative methods (e.g., semantic similarity of embeddings) might be more robust for certain tasks.

In conclusion, the paper argues that while silicon samples hold transformative potential, the field is currently "sleepwalking" into a crisis of reliability. Immediate adoption of rigorous, transparent, and context-specific development standards is required to prevent the generation of a non-replicable scientific literature.