Technical Summary: Mitigating LLM-based p-Hacking by Preregistering for the Next LLM

Problem Statement

Large Language Models (LLMs) are increasingly utilized to generate, classify, and annotate data that feeds into downstream hypothesis tests. However, this reliance introduces a severe vulnerability to p-hacking. Unlike traditional research where data collection is a distinct phase, LLM-based research allows researchers to iteratively tune prompts, decoding parameters (e.g., temperature, top-p), and output formats until a desired statistical significance is achieved. Because LLMs are always accessible, researchers can repeatedly inspect and revise analyses, effectively manufacturing significance even when no true effect exists.

This issue is exacerbated by the rise of autonomous AI scientists, which can conduct end-to-end research and engage in p-hacking without human direction. Traditional preregistration, which commits researchers to hypotheses and analysis plans before data collection, fails in this context because the "data" (LLM outputs) can be generated and manipulated immediately after the commitment. The core challenge is that existing safeguards do not transfer to LLMs, as the flexibility that makes LLMs powerful also makes them easily hackable.

Methodology: The Proposed Protocol

The authors propose a protocol to mitigate p-hacking by exploiting two key observations:

1. Non-transferability: A prompt or decoding configuration that forces one LLM to produce a desired label often fails to do so on a different model.
2. Temporal Commitment: Researchers can commit to an analysis plan before the confirmatory model exists.

The protocol consists of four steps:

1. Finalize Procedure: The researcher conducts pilot experiments using currently available models to determine experimental procedures, evaluation criteria, and analysis plans.
2. Preregister Procedure: The researcher preregisters the finalized procedures and analysis plan.
3. Preregister Eligible Models: The researcher preregisters a broad set of future eligible models (e.g., specific families from multiple providers) and defines precise inclusion/exclusion criteria (e.g., price per token, benchmark scores) to determine which future release qualifies.
4. Evaluate on Next Eligible Model: Once the next eligible LLM is released, the researcher conducts the confirmatory analysis using the preregistered procedure on this new model.

Crucially, the confirmatory model does not exist at the time of commitment, preventing the researcher from hacking against it. The protocol includes a detailed template for preregistration (e.g., on AsPredicted) that specifies prompts, decoding parameters, output validity rules, and the selection logic for the confirmatory model.

Evaluation Design

The authors evaluated the protocol using two tasks where the true effect is known to be null (by construction), allowing them to identify successful p-hacking:

• Review Classification: Estimating the proportion of AI-generated text in a corpus of 2020 peer reviews (true prevalence: 0%).
• Dietary Comparison: Estimating whether the second of two consecutive food logs has lower calories than the first (true rate: 50%).

Experimental Setup:

• Models: 20 models from four providers (OpenAI, Anthropic, Google, xAI).
• Configurations: 11 distinct configurations varying prompt structure, instruction framing, batching, and decoding parameters.
• Phases:
  • Retrospective: Evaluated 14 models released before the preregistration date (March 25, 2026).
  • Preregistration: The authors preregistered the full protocol, datasets, and 11 configurations.
  • Prospective: Evaluated the first eligible model released after preregistration (Grok 4.20) and five subsequent models.

Metrics:

• Mitigation Rate: The fraction of cases where a configuration successfully p-hacked a model but failed to p-hack the next released model under the same configuration.
• Detection Power: The ability of the protocol to detect true effects (tested by injecting known effect sizes into the dietary task) compared to a baseline of re-measuring with the same model.

Key Results

• Mitigation Effectiveness:
  • In the Review Classification task, the protocol would have blocked the transfer of p-hacking in 73.9% of cases (17 out of 23 successful hacks).
  • In the Dietary Comparison task, the protocol blocked transfer in 72.7% of cases (32 out of 44 successful hacks).
  • In the authors' own prospective experiment, 6 out of 7 configurations that hacked the pre-preregistration model failed to hack the first eligible post-preregistration model (85.7% mitigation).
• Robustness to Strategy:
  • Even when an adversary strategically selects the configuration to preregister (e.g., choosing the configuration with the highest target-label rate on the current model or the one most frequently hacked in the past), the mitigation rate remained above 60%.
  • Restricting the eligible set to a single provider (e.g., only OpenAI or only Anthropic) still yielded a mitigation rate of at least 50%, though results were heterogeneous due to smaller sample sizes.
• Detection Power: The protocol retains the ability to detect genuine effects. The detection rate for true effects using the next model was comparable to re-measuring with the same model, indicating no significant loss of statistical power.
• Time Cost: The median waiting time for the next eligible model release in the study was approximately 10.3 days, with a mean of 14.5 days.

Significance and Claims

The paper claims that preregistering for the next LLM is a viable and effective mechanism to safeguard the credibility of LLM-based research. By committing to a procedure and a set of eligible future models before the confirmatory model is released, researchers remove the ability to iteratively search for configurations that produce desired results.

Key claims include:

• Feasibility: The protocol is practical, requiring only a short wait (median ~10 days) and utilizing existing preregistration platforms.
• Generalizability: The approach works across different model providers and is robust even if researchers attempt to game the system by selecting specific configurations or providers.
• Trust: The protocol shifts LLM-based research from an open-ended search over "forking paths" to a transparent commitment, providing a verifiable signal that results were not searched for.
• Autonomous AI: The protocol is particularly relevant for autonomous AI scientists, which can be integrated into research frameworks to automatically commit to procedures before confirmatory models are available.

The authors acknowledge limitations, noting that the protocol reduces but does not eliminate p-hacking, as some configurations do propagate across models. Furthermore, the evaluation focused on binary classification tasks; future work is needed to assess effectiveness on multi-class labeling, free-form generation, and regression-style outputs. The paper concludes that while the approach requires a modest delay, it offers a demonstrably trustworthy alternative for research relying on LLM-generated data.