p-Hacking Inflates Type I Error Rates in the Error Statistical Approach but not in the Formal Inference Approach

1. Problem Statement

The paper addresses the widespread consensus that p-hacking (conducting multiple significance tests and selectively reporting only significant results without correction) inflates Type I error rates, thereby contributing to the replication crisis in science.

• The Conflict: While the Error Statistical Approach (associated with Deborah Mayo and Aris Spanos) argues that p-hacking fundamentally invalidates significance testing by inflating the "actual" error rate above the nominal level, the author argues that this view relies on a specific philosophical framework that is not universal.
• The Core Question: Does p-hacking mathematically inflate Type I error rates in all philosophical contexts of significance testing, or is this inflation specific to the Error Statistical Approach?

2. Methodology and Theoretical Framework

The paper employs a philosophical and logical analysis rather than empirical simulation. It contrasts two distinct philosophies of significance testing:

1. The Error Statistical Approach (Mayo, Spanos):

   • Focuses on the "actual" test procedure, including the researcher's hidden selection processes (e.g., data dredging, optional stopping).
   • Argues that the relevant error rate is the Familywise Error Rate (FWER) of the entire "hunting expedition" (the set of all tests conducted, reported or not).
   • Requires the "minimal severity requirement": A claim is only well-tested if the procedure rules out alternative explanations (including selection bias).
   • View on P-hacking: P-hacking is a "biasing selection effect" that alters the sample space, making the nominal p-value (e.g., 0.05) invalid because the "actual" probability of a false positive is much higher (e.g., $1 - (1-0.05)^k$).

2. The Formal Inference Approach (Rubin):

   • Focuses strictly on formally reported statistical inferences.
   • Distinguishes between the context of discovery (psychological/historical process of hypothesis generation) and the context of justification (logical evaluation of the reported test).
   • Defines Type I error rates as hypothetical, subjunctive conditional probabilities based on the specific null hypothesis and sampling distribution as reported.
   • View on P-hacking: P-hacking is part of the context of discovery. If a researcher reports an individual inference (testing a single hypothesis $H_{0,2}$), the relevant error rate is the nominal $\alpha$ for that specific test, regardless of unreported tests ($H_{0,1}$) that preceded it.

3. Key Contributions and Arguments

A. The Incommensurability of Error Rates

The author argues that comparing the "actual" FWER (from the error statistical view) with the nominal rate of a reported individual inference is a category mistake.

• Logical Fallacy of Division: The FWER applies to the intersection null hypothesis ($H_{0,1} \cap H_{0,2}$). It represents the probability of rejecting the entire family of hypotheses. It does not license inferences about individual members of that family (e.g., $H_{0,2}$ alone).
• Incommensurability: The error rate for an individual test ($H_{0,2}$) is based on a specific sampling distribution. The FWER is based on a different, broader sampling distribution. One cannot claim the individual error rate is "inflated" by the FWER because they refer to different logical propositions.

B. The Role of "Actual" vs. "Formal" Procedures

• Error Statistical View: The "actual" procedure includes the researcher's hidden steps. Therefore, the sample space is expanded to include all possible outcomes of the "hunting expedition," inflating the error rate.
• Formal Inference View: The "actual" procedure is irrelevant to the validity of the reported inference. The relevant procedure is the formally specified one. If a researcher reports an individual test of $H_{0,2}$, the inference is licensed by the nominal $\alpha$ of that specific test. The unreported tests are logically distinct and do not alter the mathematical properties of the reported test.

C. Rejection of the "Minimal Severity Requirement" for Validity

The paper argues that the minimal severity requirement (the need to rule out systematic error and selection bias to validate an inference) is not necessary for the statistical validity of a Type I error rate.

• Type I error rates represent random sampling error under a set of implicatory assumptions (i-assumptions).
• Even if the "actual" assumptions are violated (e.g., due to p-hacking or fraud), the nominal error rate remains a valid mathematical statement about what would happen if the assumptions held.
• Analogy: A sharpshooter painting a target around bullet holes (Texas Sharpshooter fallacy) commits fraud. However, the mathematical probability of hitting that specific painted target by chance remains valid as a hypothetical statement. The fraud invalidates the substantive claim of skill, not the statistical calculation of the probability.

D. Nondisclosure of Null Results

• Error Statistical View: Hiding null results inflates the FWER and biases the evidential weight.
• Formal Inference View: Hiding null results does not inflate the Type I error rate of the reported test. However, it can bias substantive inferences if the null results are viewed as "evidence of absence" and are theoretically relevant. The solution is not statistical correction (like preregistration) but critical rational discussion among peers to identify theoretically relevant omitted tests.

4. Results and Conclusions

1. P-hacking does not inflate Type I error rates in the Formal Inference Approach.

   • In this framework, the reported nominal error rate (e.g., 0.05) is valid because it applies strictly to the formally reported individual inference.
   • The "inflation" argument only holds if one accepts the Error Statistical premise that the "actual" hidden procedure defines the relevant sample space.

2. The Replication Crisis is not primarily a statistical error problem.

   • The author argues that replication failures are not caused by inflated Type I error rates (which remain at nominal levels in the formal view).
   • Instead, the crisis stems from unrecognized interpretational errors (systematic errors, hidden moderators, theoretical flaws) that are not captured by statistical models.
   • Auditing and preregistration cannot solve "unknown unknowns" (errors researchers haven't thought to investigate).

3. Implications for Scientific Practice:

   • Preregistration: While useful for error statisticians to define the "actual" procedure, it is not logically necessary for the validity of Type I error rates in the formal inference approach.
   • Focus Shift: The solution to the replication crisis requires an epistemological shift toward acknowledging the "finitude and fallibility of knowledge" (Popper) and engaging in rigorous theoretical critique, rather than relying solely on statistical corrections for selection bias.

5. Significance

This paper provides a rigorous philosophical defense of the Formal Inference Approach, challenging the dominant narrative that p-hacking universally invalidates statistical significance.

• Theoretical Impact: It clarifies that "Type I error inflation" is not a mathematical inevitability but a consequence of adopting the Error Statistical philosophy.
• Practical Impact: It suggests that the focus on p-hacking as a statistical sin may be misplaced. Instead, the scientific community should focus on the substantive validity of hypotheses and the theoretical relevance of reported results, recognizing that statistical significance is a conditional logical statement, not a guarantee of truth in the real world.
• Replication Crisis: It reframes the replication crisis as a problem of epistemology and theory rather than a failure of statistical methodology, arguing that no amount of statistical auditing can eliminate the "unknown unknowns" that drive replication failures.