Technical Summary: From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology

Problem Statement

Large Language Models (LLMs) are increasingly utilized in psychological research both as tools (e.g., for coding text or generating stimuli) and as objects of inquiry (e.g., to simulate human behavior or model cognitive processes). However, a critical methodological gap exists: researchers frequently apply human-validated psychometric instruments (inventories, questionnaires, behavioral tasks) to LLMs without establishing that the outputs are reliable or interpretable.

This practice rests on untested assumptions: that human-validated instruments remain valid for nonhuman systems, that surface-similar outputs reflect similar underlying processes, and that statistical patterns in model responses carry the same inferential weight as human data. The paper argues that these assumptions are empirically fragile. Evidence suggests that LLM responses are highly sensitive to trivial prompt perturbations (e.g., punctuation, spacing, option order) and training artifacts (e.g., sycophancy, alignment biases), leading to "measurement phantoms"—statistical regularities mistaken for genuine psychological phenomena. Consequently, claims regarding LLM personality, theory of mind, or moral reasoning often lack the necessary psychometric and causal foundations.

Methodological Framework

The paper proposes a Dual-Validity Framework that integrates two distinct methodological traditions in psychology:

1. The Psychometric Tradition: Focuses on measurement quality and the meaning of scores (Construct Validity). It asks whether instruments measure intended constructs through evidence from content, response processes, internal structure, relations to other variables, and consequences.
2. The Causal Inference Tradition: Focuses on warranted conclusions about cause and effect. It evaluates threats to internal, external, construct, and statistical conclusion validity.

The framework posits that LLM research requires the integration of both traditions because the same LLM output often serves as both a measurement indicator (what the model "is") and an experimental outcome (what the model "does" under manipulation).

The Hierarchy of Scientific Ambition

The paper establishes a hierarchy of claims, where evidentiary burdens scale with scientific ambition:

• Tool Use: Requires task validation (accuracy, robustness to prompt variation).
• Behavioral Characterization (Machine Psychology): Requires psychometric validation (reliability, construct validity) to support descriptive claims about model properties (e.g., "The model exhibits trait X").
• Human Simulation: Requires behavioral correspondence (reproducing human response distributions) plus the validation requirements of behavioral characterization.
• Cognitive Modeling: Requires the strongest burden: evidence that the model's computational processes illuminate human cognitive mechanisms, not just that outputs resemble human answers.

Key Contributions and Diagnostic Findings

1. Reliability Failures as a Precondition for Validity

The paper details three primary sources of instability that compromise the reliability of LLM measurements, thereby placing a hard ceiling on validity:

• Training Artifacts: Post-training procedures like Reinforcement Learning from Human Feedback (RLHF) induce sycophancy (user-directed agreement) and acquiescence bias (endorsing contradictory items). Training data contamination leads to memorization rather than latent trait expression.
• Prompt Hypersensitivity: LLMs exhibit disproportionate sensitivity to non-semantic variations (e.g., changing "Case 1" to "(A)" or altering punctuation). This violates parallel-forms reliability, with accuracy shifts of up to 76 percentage points across prompt formats.
• Generation Instability: Stochastic sampling (temperature settings), context window accumulation (where recent turns overshadow earlier ones), and version drift (updates changing response patterns) introduce unreliability that is absent in biological systems.

2. Diagnostic Map of Validity Threats

The paper provides a diagnostic taxonomy (Table 1) mapping validity threats to their computational mechanisms:

• Psychometric Threats:
  • Content: Domain undersampling (e.g., using only trolley problems for moral reasoning) and training-data contamination.
  • Response Processes: Mechanistic substitution (next-token prediction vs. psychological deliberation) and architectural artifacts (attention biases).
  • Internal Structure: Structural misfit (factor models failing to replicate human dimensionality) and factorial collapse (alignment-induced agreeableness compressing variance).
  • Relations to Other Variables: Nomological instability (correlations shifting across models/parameters) and behavioral-report disconnects.
  • Consequences: Bias reification (interpreting corpus biases as stable traits) and inappropriate application.
• Causal Inference Threats:
  • Internal: Sampling-parameter confounding (e.g., temperature covarying with manipulation), prompt confounding (language vs. culture), and version-drift confounding.
  • External: Generalization failure across models/prompts and population mismatch (projecting model outputs to humans without validation).
  • Construct: Manipulation-validity failure (altering token probabilities rather than psychological mechanisms) and outcome-validity mismatch (measuring linguistic fluency rather than capacity).
  • Statistical Conclusion: Non-independence of model outputs (treating multiple completions as independent participants) and variance restriction.

3. The Necessity of Computational Analogs

The paper argues that applying human measures to LLMs and checking for matches is the wrong starting point. Instead, researchers must develop computational analogs of psychological constructs. These are reconceptualizations that preserve a construct's theoretical role while acknowledging mechanistic differences (e.g., defining "anxiety" in an LLM as elevated uncertainty markers and negative-valence language, rather than embodied threat responses).

Results and Evidence

The paper synthesizes emerging evidence demonstrating that current practices often fail to meet validity standards:

• Reliability: Studies show LLM personality scores can shift drastically based on prompt wording or temperature settings, and models often endorse contradictory statements (e.g., "I am an introvert" and "I am an extrovert").
• Construct Validity: Factor analyses of LLM responses to the Big Five Inventory often fail to reproduce the five-factor structure found in humans. Furthermore, "success" in theory-of-mind tasks often reflects rigid heuristics or training data retrieval rather than genuine mentalistic understanding.
• Causal Inference: Experiments manipulating prompts to test cultural or ethical differences often suffer from uncontrolled confounds (e.g., language differences confounded with cultural identity), rendering causal claims ambiguous.

Significance and Claims

The paper claims that the field of "AI psychology" is currently at risk of producing measurement phantoms due to the uncritical transfer of human psychometric standards to stochastic, non-biological systems. Its primary significance lies in:

1. Establishing a Dual-Validity Standard: Arguing that robust AI psychological research must simultaneously satisfy psychometric requirements (what the score means) and causal inference requirements (what the study warrants).
2. Defining Evidentiary Hierarchies: Clarifying that stronger claims (e.g., cognitive modeling) require a cumulative foundation of reliability and validity evidence that is currently missing in much of the literature.
3. Reframing the Research Agenda: Shifting the focus from "Do LLMs have human traits?" to "What computational mechanisms produce human-like patterns, and how can we measure them on their own terms?"

The paper concludes that progress depends not on assuming human measures apply automatically, but on developing computational analogs of constructs and experimental designs that control for computational confounds. It calls for a coordinated reform in research practice, including preregistration of measurement approaches, the development of validated prompt repositories, and statistical standards tailored to the non-independence and stochastic nature of LLM data.