Six Fallacies in Substituting Large Language Models for Human Participants

Technical Summary: Six Fallacies in Substituting Large Language Models for Human Participants

Problem Statement
The rapid advancement of Large Language Models (LLMs) has sparked a proposal within behavioral and psychological sciences to substitute human participants with "silicon samples" or "simulated participants" for data collection. Proponents suggest that LLMs can mirror average human responses in tasks ranging from moral reasoning to language perception, offering a resource-efficient alternative to human data collection. However, this "replacement perspective" lacks critical methodological and theoretical scrutiny. The paper identifies a fundamental epistemic risk: researchers may conflate the statistical capabilities of LLMs with human cognition, leading to invalid inferences about human psychology. The core problem is the lack of a framework to distinguish between LLMs as simulators of linguistic behavior versus replacements for human participants, a distinction necessary to preserve research validity.

Methodology
This paper employs a critical conceptual analysis rather than empirical experimentation. The author systematically deconstructs the "replacement perspective" by identifying and analyzing six specific interpretive fallacies that arise when LLMs are treated as human substitutes. The analysis draws upon:

• Computational Theory: Examining the architecture of LLMs (autoregressive token prediction, transformer mechanisms, RLHF) versus biological intelligence (embodied experience, evolutionary constraints).
• Philosophical Frameworks: Utilizing concepts such as the "Chinese Room" argument, multiple realizability, and the distinction between functional and mechanistic equivalence.
• Existing Empirical Literature: Synthesizing findings from recent studies (e.g., Aher et al., 2023; Binz & Schulz, 2023a; Dillion et al., 2023) that demonstrate both the capabilities and limitations of LLMs in psychological tasks.
• Validity Theory: Applying standards of research validity (internal, external, and construct validity) to evaluate the threats posed by each fallacy.

Key Contributions
The paper's primary contribution is the identification and detailed exposition of six interpretive fallacies that undermine the validity of using LLMs as human replacements:

1. Token Prediction as Human Intelligence Fallacy: The error of equating statistical pattern matching (next-token prediction) with genuine human understanding. LLMs lack embodied experience, sensorimotor grounding, and consciousness, operating instead as "ungrounded" statistical models. They capture linguistic form but not the communicative intent or meaning derived from lived experience.
2. The Average Human Fallacy: The assumption that LLM outputs represent the "average" human. The paper argues that LLMs are not representative samples due to training data biases (predominantly WEIRD populations), temporal recency bias, and the influence of Reinforcement Learning from Human Feedback (RLHF), which optimizes for helpfulness and safety rather than reflecting raw human diversity or idiosyncrasies.
3. Alignment as Explanation Fallacy: The misconception that surface-level alignment between LLM and human outputs implies shared cognitive mechanisms. The paper emphasizes "multiple realizability," noting that different systems (statistical vs. biological) can produce identical outputs via distinct internal processes. Output similarity does not prove mechanistic equivalence.
4. Anthropomorphism Fallacy: The tendency to attribute human mental states (beliefs, intentions, emotions) to LLMs based on fluent language generation. This leads to the "language user illusion," where researchers misinterpret statistical artifacts as expressions of an inner mental life, invalidating constructs that presuppose consciousness.
5. Identity Essentialization Fallacy: The practice of prompting LLMs with static demographic labels (e.g., "Black woman," "American") as if these categories represent homogeneous, fixed traits. This reduces fluid, intersectional, and contextual human identities to stereotypes, failing to capture the nuance of real-world identity negotiation.
6. Substitution Fallacy: The presumption that LLM-generated data can directly replace human evidence without validation. This ignores the static nature of training data (knowledge cutoffs), the risk of closed-loop information systems (where models amplify past biases), and the inability of models to adapt to real-time societal changes.

Results and Analysis
The analysis demonstrates that while LLMs can function as pragmatic simulation tools (Level 1 equivalence), they fail to meet the criteria for explanatory inference (Level 2) or phenomenological attribution (Level 3).

• Technical vs. Conceptual Limitations: The paper distinguishes between technical limitations (e.g., data bias, prompt sensitivity), which may be addressable through engineering, and conceptual limitations (e.g., lack of grounding, agency, and subjectivity), which are intrinsic to the current paradigm of statistical text prediction.
• Validity Threats: Each fallacy is shown to compromise specific aspects of research validity. For instance, the "Average Human Fallacy" threatens external validity (generalizability), while the "Alignment as Explanation Fallacy" threatens internal validity (causal inference).
• Safeguards: For each fallacy, the paper proposes specific methodological safeguards, including:
  • Validating LLM outputs against human data.
  • Using contextualized, intersectional prompting rather than essentialist labels.
  • Distinguishing between functional and mechanistic equivalence in reporting.
  • Implementing sequential research designs where LLMs are used for hypothesis generation but findings are validated with human participants.

Significance and Claims
The paper claims that the "replacement perspective" is fundamentally flawed because it misidentifies the nature of LLMs. LLMs are not "minds" but "neural language simulators" that reveal latent psychological dynamics embedded in text, rather than the minds themselves.

• Reframing the Role of AI: The author argues for a shift in conceptualizing LLMs from substitutes to complementary tools. They are valuable for rapid prototyping, role-playing, and computational modeling, provided their outputs are rigorously validated against human data.
• Preserving Research Integrity: By exposing these fallacies, the paper aims to prevent the erosion of research validity in behavioral sciences. It warns that uncritical substitution risks creating epistemic echo chambers and misrepresenting human cognition.
• Modest Conclusion: The paper does not claim that LLMs will never be able to simulate human behavior more accurately, nor does it deny their utility. Instead, it asserts that current LLMs, and those foreseeable under the current statistical paradigm, cannot replace human participants for theoretical understanding of the human mind due to fundamental differences in grounding and consciousness. The ultimate goal is to leverage LLMs productively while respecting the ontological and epistemological distinctions between machine intelligence and human thought.