Technical Summary: A Validity-Guided Workflow for Robust Large Language Model Research in Psychology

Problem Statement

The integration of Large Language Models (LLMs) into psychological research has accelerated, with models utilized as research tools, evaluation targets, human simulators, and cognitive models. However, recent evidence indicates severe measurement unreliability in this domain. Phenomena such as personality assessments collapsing under factor analysis, moral preferences reversing with punctuation changes, and theory-of-mind accuracy fluctuating with trivial rephrasing suggest the presence of "measurement phantoms"—statistical artifacts masquerading as psychological phenomena.

Current research often lacks the validity evidence required to support claims about LLM psychology. These methodological failures violate fundamental psychometric principles (stability and construct validity) and causal inference standards (internal and external validity), undermining the reproducibility and robustness of the field. The core problem is that researchers frequently make claims about LLM psychological properties without establishing that their instruments measure intended constructs rather than superficial formatting cues or training data artifacts.

Methodology: The Dual-Validity Framework and Six-Stage Workflow

To address these issues, the paper proposes a measurement-first workflow grounded in a Dual-Validity Framework. This framework integrates two distinct traditions:

1. Psychometric Validation: Ensuring instruments measure intended constructs (reliability, internal structure, convergent/discriminant validity).
2. Causal Inference: Ensuring observed relationships represent genuine causal phenomena rather than computational confounds (internal, external, and statistical conclusion validity).

The workflow operationalizes these standards into six sequential yet iterative stages:

Stage 1: Define Research Goal

Researchers must explicitly classify their ambition into one of four categories, each with escalating validity requirements:

• LLMs as Research Tools: Functional performance claims (e.g., classification accuracy).
• LLMs as Evaluation Targets: Descriptive claims about stable behavioral patterns (e.g., model "personality").
• LLMs as Human Simulators: Comparative claims regarding statistical parallelism with human populations.
• LLMs as Cognitive Models: Mechanistic claims about structural analogies to human cognition.
  Decision: The classification dictates the necessary validation evidence (e.g., cognitive models require mechanistic investigation, while tools require functional reliability).

Stage 2: Develop and Validate the Computational Instrument

This stage diverges based on the research goal:

• Pathway A (Research Tools): Focuses on task-specific validity (agreement with human coders, accuracy against gold standards, or content validity for stimulus generation).
• Pathway B (Evaluation Targets, Simulators, Cognitive Models): Requires comprehensive psychometric validation adapted for computational realities:
  • Content Validity: Reconceptualizing human constructs for disembodied systems (e.g., defining "conscientiousness" as systematic response organization rather than self-discipline) and generating item pools while avoiding training data contamination and linguistic confounds.
  • Reliability Assessment: Establishing measurement stability via test-retest consistency (characterizing response variance), parallel forms reliability (testing sensitivity to syntactic variations), and parameter stability.
  • Construct Validity: Analyzing internal structure (factor analysis), investigating response processes (using chain-of-thought or ablation studies to distinguish reasoning from pattern matching), and establishing convergent/discriminant validity through nomological networks.
  • Consequential Evidence: Assessing behavioral predictivity and social/ethical implications.

Stage 3: Design Experiment

With validated instruments, researchers design experiments to test causal hypotheses. Key requirements include:

• Operationalization: Defining manipulations that instantiate theoretical causes rather than mere linguistic correlates.
• Threat Control: Addressing internal validity threats (prompt-level confounds, technical confounds like model updates, context accumulation) and external validity threats (generalizability across models and human populations).
• Statistical Conclusion Validity: Accounting for non-independent observations (clustering) and the "correct answer" effect (reduced variance).
• Pre-registration: Specifying design, parameters, and analysis plans to prevent HARKing and p-hacking.

Stage 4: Execute and Document

Execution requires faithful adherence to the pre-registered protocol with transparent documentation of deviations. Crucial elements include:

• Environment Specification: Recording exact model versions, API parameters (temperature, top_p), and data collection dates.
• Data Preservation: Storing full raw prompts, unprocessed outputs, and metadata to ensure future auditability despite model deprecation.

Stage 5: Analyze and Interpret

Analysis must adapt to the non-independence of LLM data:

• Non-Independence Handling: Using aggregation, multilevel modeling, or cluster-level bootstrap resampling to account for within-model clustering.
• Robustness Checks: Testing findings against technical perturbations (parameter changes) and conceptual variations (task logic changes) to rule out measurement phantoms.
• Effect Calibration: Interpreting effect sizes cautiously, acknowledging that high statistical power may result from low variance rather than strong effects.

Stage 6: Report and Reconceptualize

Reporting must be transparent, adhering to guidelines like TRIPOD-LLM or MI-CLEAR-LLM. Claims must be constrained to demonstrated boundaries, avoiding anthropomorphic language unless operationalized. The workflow encourages using findings to refine theories and constructs, potentially abandoning human-centric labels in favor of computationally grounded descriptions.

Key Contributions

1. Dual-Validity Framework: A theoretical integration of psychometrics and causal inference specifically tailored for LLM research, distinguishing between measurement artifacts and genuine phenomena.
2. Six-Stage Workflow: A practical, operationalized guide that scales validity requirements to research ambition, preventing the accumulation of validity threats.
3. Reconceptualization of Constructs: A methodological approach to adapting human psychological constructs for disembodied statistical models, moving beyond naive anthropomorphism.
4. Technical Protocols: Specific recommendations for handling LLM-specific challenges, including API-based reliability testing, prompt perturbation, and statistical methods for clustered data.
5. Illustrative Example: A demonstration of the workflow applied to measuring "LLM selfhood," showing how to distinguish stable linguistic patterns from measurement artifacts.

Results and Claims

The paper does not present new empirical data or experimental results of its own. Instead, it synthesizes existing evidence of methodological failures (e.g., personality assessments collapsing, moral judgments fluctuating with formatting) to argue that the field is currently plagued by "measurement phantoms."

The paper claims that adopting this workflow will:

• Distinguish genuine computational phenomena from statistical artifacts.
• Provide a path toward building a robust empirical foundation for AI psychology.
• Facilitate the transition from applying human metaphors to characterizing genuine computational phenomena.
• Enable the creation of shared repositories for validated computational instruments, fostering cumulative science.

Significance

The paper argues that methodological rigor is not an obstacle to discovery but the only path toward it. By introducing "friction" into the research process—slowing down the "prompt-and-publish" cycle in favor of front-loaded validation—the workflow aims to transform isolated studies into a cumulative science. It calls for a cultural shift in the field, prioritizing replicability and rigor over rapid discovery. The ultimate significance lies in establishing a standard for "validated computational instruments" that can track model evolution over time and prevent the reification of measurement artifacts as psychological truths.