Preregistration for Experiments with AI Agents

Technical Summary: Preregistration for Experiments with AI Agents

Problem Statement
The paper identifies a critical methodological gap in the emerging paradigm of "in silico" behavioral experiments, where Large Language Models (LLMs) and autonomous AI agents serve as participants in studies of cognition, decision-making, and social dynamics. While these experiments offer unprecedented scalability and cost-efficiency compared to human subjects research, they inherit and amplify the "researcher degrees of freedom" that drove the replication crisis in the social sciences.

In traditional human research, high marginal costs (recruitment, compensation, time) impose natural friction against exhaustive exploration of experimental specifications. In contrast, AI experiments possess a unique risk profile: high specification flexibility combined with near-zero marginal costs. Researchers can rapidly iterate through a vast combinatorial space of configurations—varying model families, prompt wording, decoding parameters (temperature, top-p), seeds, retry policies, and parsing rules. This creates a "garden of forking paths" where researchers, intentionally or inadvertently, can traverse thousands of defensible specifications until a desired result emerges. Because these iterations often leave no visible trace in final publications, the field risks accumulating literature where findings reflect undisclosed optimization processes rather than robust phenomena.

Methodology and Taxonomy
The paper does not present new experimental data but rather constructs a systematic taxonomy of researcher degrees of freedom specific to AI agent experiments. The authors map these vulnerabilities onto the "p-hacking" dynamics identified in human subjects research, categorizing them across the full experimental pipeline:

1. Model Selection: Choosing among dozens of models with varying capabilities and alignment tuning.
2. Prompt Engineering: Iterative refinement of system messages, user instructions, and few-shot examples, where minor wording changes act as high-dimensional treatment manipulations.
3. Sampling and Decoding: Adjusting stochastic parameters (temperature, top-p) and seeds that alter response content and refusal behaviors.
4. Inference Budget: Varying token limits, turn counts, and tool calls, which can fundamentally change agent strategies.
5. Response Processing: Decisions on how to handle refusals, malformed outputs, or outliers (e.g., exclusion, imputation, or retrying).
6. Outcome Operationalization: Selecting specific metrics or evaluation prompts (including LLM-based judges) post-hoc.

To illustrate the magnitude of this problem, the authors reference a simulation (detailed in the paper's Appendix) examining anchoring effects across 2,430 specifications. The simulation demonstrated that by selecting different paths through the specification space, a researcher could conclude that LLMs exhibit strong human-like anchoring, no anchoring, or even reverse anchoring. This highlights the "illusion of robustness," where a finding appears generalizable simply because the researcher navigated the multiverse of specifications to find a supporting path.

Key Contributions
The paper's primary contribution is the proposal to extend preregistration—a practice central to improving credibility in human subjects research—to experiments with AI agents. The authors introduce a tailored preregistration template designed to constrain flexibility while preserving necessary exploratory freedom. Key components of this template include:

• Computational Environment Specification: Requiring exact model identifiers (including version checkpoints), generation parameters, inference budgets, and API access dates to ensure reproducibility.
• Prompt Transparency: Mandating verbatim disclosure of all prompt text (system and user messages) rather than summaries, as subtle formatting changes significantly impact behavior.
• Pilot Testing Disclosure: Requiring researchers to document the history of prompt iterations, models tested during development, and whether preliminary results influenced final hypotheses. This addresses the invisibility of specification search.
• Confirmatory vs. Exploratory Distinction: Explicitly labeling analyses as either hypothesis-testing (confirmatory) or hypothesis-generating (exploratory). For confirmatory studies, the template requires pre-specification of primary outcomes, statistical tests, and decision rules.
• Robustness and Generalization Protocols: If researchers intend to test robustness across models or prompt variants, they must define the full space of specifications and aggregation rules in advance, reporting results from the entire space rather than selecting a subset.
• Staged Designs: Accommodating adaptive designs (e.g., using pilot data to calibrate difficulty) by requiring pre-specified decision rules for subsequent stages.
• Transparency Commitments: Mandating the sharing of raw model outputs, processed data, and analysis code, along with an attestation that data collection has not yet begun.

Results and Claims
The paper does not report empirical results from a new study but synthesizes evidence from the social sciences and machine learning literature to argue for a shift in norms. The authors claim that:

1. The Risk is Systemic: The combination of high flexibility and low cost in AI experiments creates incentives for specification search that are more severe than in human subjects research.
2. Current Norms are Insufficient: Unlike traditional ML benchmarking, which relies on fixed test sets, behavioral experiments with AI agents often lack analogous safeguards, leading to a literature where reported findings may be artifacts of undisclosed optimization.
3. Preregistration is the Solution: By requiring advance commitment to hypotheses, methods, and analysis plans, preregistration can constrain the "garden of forking paths," distinguishing between confirmatory tests and exploratory searches.
4. Feasibility: The proposed template balances rigor with practicality, allowing for exploratory work as long as it is clearly labeled and separated from confirmatory claims.

Significance
The paper argues that the machine learning community has a unique opportunity to "build credibility into this research paradigm from the outset" rather than retrofitting safeguards after a crisis, as the social sciences did. By institutionalizing preregistration, the field can ensure that findings regarding AI behavior—such as biases, reasoning patterns, and alignment properties—are robust and trustworthy. This is particularly critical as AI agents increasingly assume autonomous roles in high-stakes domains (e.g., negotiation, infrastructure management), where understanding their behavior is a research priority in its own right. The authors call on conferences, journals, and funding agencies to make preregistration standard practice, thereby transforming the research process from an opaque journey through a multiverse of specifications into a legible, cumulative science.