📊 statistics

Precision and Decisiveness as Goals: Reliable Sequential Hypothesis Testing with a Dual Stopping Criterion

This paper introduces "Decisive Precision is the Goal" (DPitG), a new sequential hypothesis testing framework that simultaneously satisfies precision and decisiveness criteria to eliminate the false-positive risks of early stopping and the high inconclusive rates of precision-only methods, thereby offering a reliable, pre-registerable approach for obtaining definitive statistical verdicts.

Original authors: Eyal A. Kazin

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eyal A. Kazin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Detective Hunt: When to Stop Looking

Imagine you are a detective trying to solve a mystery: Is a new coin fair, or is it rigged? In the world of science, researchers are constantly playing this game. They collect data—like flipping a coin or testing a new medicine—to see if their hunch is right. But here is the tricky part: How many times do you need to flip the coin before you can confidently say, "Okay, I know the answer!"?

If you stop too early, you might catch a lucky streak of heads and wrongly accuse a fair coin of being rigged. This is like a detective arresting an innocent person because they saw them running once, without checking the whole story. On the other hand, if you keep flipping the coin forever, you might never finish your report, wasting time and money. Scientists have been trying to find the perfect "stop rule" for decades. They want a method that is fast enough to be useful but careful enough to be right. This paper dives into a specific branch of statistics called Sequential Hypothesis Testing, which is all about making these decisions as data rolls in, rather than waiting until a pre-set number of flips is reached. It focuses on two main ideas: Precision (how tight your estimate is) and Decisiveness (whether you can actually say "Yes" or "No" to the question).

The Problem with "Peeking" and the "Maybe" Trap

The paper starts by looking at two popular ways scientists try to solve this puzzle, and it finds that both have a fatal flaw.

The first method is like a detective who stops the moment they see a clue that looks suspicious. If the data looks like the coin is rigged, they stop and declare it guilty. If it looks fair, they stop and declare it innocent. The problem? If you peek at the data too early, you might catch a random fluke. The paper shows that this "stop-as-soon-as-you-know" approach leads to a systematic error: it falsely accuses fair coins about 6.3% of the time. It's like convicting an innocent person just because they happened to look nervous once.

The second method, called "Precision is the Goal" (PitG), is much more careful. It says, "Don't stop until your estimate is super precise, no matter what the result looks like." Imagine a detective who refuses to make an arrest until they have gathered enough evidence to be 100% sure of the facts. This stops the false accusations, but it creates a new problem: The "Maybe" Trap. When the coin is actually fair, this method often stops with a result that is precise but still stuck in the middle, unable to say "Guilty" or "Innocent." In the simulations run by the author, this method ended up saying "I don't know" 62% of the time when the coin was actually fair. That's a lot of unsolved cases!

The New Hero: "Decisive Precision"

Enter the paper's new solution: Decisive Precision is the Goal (DPitG).

Think of DPitG as a detective who has a strict checklist. They will not stop the investigation until two conditions are met at the exact same time:

Precision: The evidence must be sharp and clear (the "HDI" width must be smaller than a target, like 0.08).
Decisiveness: The evidence must clearly point to either "Guilty" or "Innocent" (the result must be fully outside or fully inside a "Region of Practical Equivalence," or ROPE).

The paper ran massive simulations—5,000 different coin-flipping experiments—to see how this new detective compares to the old ones.

Here is what they found:

The Old "Peeking" Detective (HDI+ROPE): Was fast but made mistakes. It falsely accused fair coins 6.3% of the time.
The "Maybe" Detective (PitG): Was very careful but gave up too often. It ended with an "Inconclusive" verdict 62.1% of the time when the coin was fair.
The New DPitG Detective: It was the best of both worlds. It reduced the "Inconclusive" rate from 62.1% down to just 2.2%. Even better, it kept the false accusation rate at 0%.

The Cost of Being Right

You might wonder, "If DPitG is so perfect, why doesn't everyone use it? Is it too slow?"

The paper shows that DPitG does require a little more patience. In the simulations, the new method needed about 4.7% more coin flips on average compared to the "Maybe" detective to reach a final answer. However, this small cost bought a massive improvement in reliability. While the "Maybe" detective was stuck saying "I don't know" in most cases, DPitG was able to give a clear, correct answer almost every time.

The author also provided a "magic formula" (a closed-form planning equation) that researchers can use to calculate exactly how many samples they need before they start, based on how precise they want to be. They even built an online calculator and shared their code so anyone can try it out.

Why This Matters

This isn't just about coins. The paper explains that this method works for any kind of data, from testing if a new drug works to checking if a website change improves sales. The big takeaway is that we can have our cake and eat it too: we can demand high-quality, precise evidence and get a clear answer, without falling into the trap of false accusations or endless indecision.

The paper concludes that DPitG is the method of choice whenever a reliable verdict is needed, provided the study has the budget to collect at least the minimum number of samples required to reach that precision. It's a tool for scientists who want to be honest about their data, refusing to declare a winner until the evidence is truly ready to speak.

Technical Summary: Precision and Decisiveness as Goals in Sequential Hypothesis Testing

1. Problem Statement

Sequential hypothesis testing offers flexibility over fixed-sample designs by allowing data evaluation as it accumulates. However, a critical tension exists between stopping rules (when to stop collecting data) and decision criteria (how to interpret the data).

The Coupling Problem: Widely used methods, such as the HDI+ROPE algorithm (Kruschke, 2011, 2013, 2015), couple the stopping rule directly to the decision criterion. The algorithm stops as soon as the posterior Highest Density Interval (HDI) falls entirely outside or inside a Region of Practical Equivalence (ROPE). While this allows for positive acceptance of the null hypothesis (unlike Null Hypothesis Significance Testing), it risks confirmation bias through "early peeking." A wide HDI on an unrepresentative early sample can satisfy the decision criterion prematurely, incurring a systematic false-positive cost.
The Decoupling Problem: To eliminate bias, the "Precision is the Goal" (PitG) method (Kruschke, 2015) decouples the stopping rule from the decision. It halts only when a target posterior precision (HDI width, $\omega_{goal}$ ) is reached, regardless of the hypothesis outcome. While this eliminates early-peeking bias, it introduces a new failure mode: when the null hypothesis is true (or practically so), PitG frequently yields inconclusive results (the HDI straddles the ROPE) rather than correctly accepting the null. This high inconclusive rate limits its utility in fields requiring rigorous confirmation of the null, such as bioequivalence trials or software regression testing.

The paper identifies a gap: a method is needed that maintains the bias-free properties of precision-based stopping while ensuring a high rate of decisive outcomes, particularly when the null is true.

2. Methodology

The author proposes a new algorithm, "Decisive Precision is the Goal" (DPitG), which modifies the PitG framework by requiring two conditions to be met simultaneously before stopping:

Precision Condition: The width of the posterior HDI ( $\omega_{HDI}$ ) must be less than or equal to a pre-specified target ( $\omega_{goal}$ ).
Decisiveness Condition: The HDI must be fully inside (Accept) or fully outside (Reject) the ROPE.

Core Components

Shared Decision Rule: All three algorithms (HDI+ROPE, PitG, DPitG) use an identical decision rule based on the relationship between the 95% HDI and the ROPE boundaries:
- Accept Null: HDI $\subset$ ROPE.
- Reject Null: HDI $\cap$ ROPE = $\emptyset$ .
- Inconclusive: HDI straddles ROPE.
Stopping Criteria Differences:
- HDI+ROPE: Stops immediately upon a decisive decision (coupled).
- PitG: Stops immediately upon reaching $\omega_{goal}$ (decoupled), then checks for a decision.
- DPitG: Continues sampling until both $\omega_{HDI} \le \omega_{goal}$ AND a decisive decision is possible.

Theoretical Framework

The paper derives a closed-form planning formula for the expected sample size ( $N_{goal}$ ) required to achieve a precision goal under a normal approximation (Central Limit Theorem):
$N_{goal} \propto \frac{V}{\omega_{goal}^2}$
where $V$ is the observation variance. For a Bernoulli parameter $\theta$ , $V \propto \theta(1-\theta)$ . This formula allows researchers to pre-specify the minimum budget required to achieve a desired precision, serving as a conservative upper bound when $\theta_{null} = 0.5$ .

Experimental Design

The methods were evaluated using synthetic data simulations based on Bernoulli trials (single-group binary data):

Scale: $M = 5,000$ independent sequences of up to $N_{max} = 1,500$ trials.
Parameters: $\theta_{null} = 0.5$ , ROPE = $[0.45, 0.55]$ ( $\Delta_{ROPE} = 0.1$ ), and $\omega_{goal} = 0.08$ .
Scenarios: Tested across a range of true parameters ( $\theta_{true}$ ) from 0.5 to 0.8, including the "fair coin" case where $\theta_{true} = \theta_{null}$ .

3. Key Results

The simulations demonstrate that DPitG successfully balances the trade-off between bias, conclusiveness, and sampling cost.

Performance in the "Fair Coin" Case ( $\theta_{true} = 0.5$ )

HDI+ROPE: Achieved a high conclusiveness rate (98.2%) but incurred a 6.3% false-positive rate (incorrectly rejecting the null).
PitG: Achieved zero false positives but suffered a 62.1% inconclusive rate, failing to provide a verdict in the majority of cases where the null was true.
DPitG: Achieved a 97.8% conclusiveness rate with zero false positives in the fair coin scenario.
- Cost: The median sampling cost for DPitG was only 4.7% higher than PitG.
- Efficiency: While PitG stops exactly at $N_{goal}$ , DPitG stops at $N_{goal}$ for a substantial fraction of experiments (those where precision and decisiveness align early) but extends sampling only when necessary to resolve indecisive states.

Sensitivity and General Trends

Robustness to $\omega_{goal}$ : As the precision goal $\omega_{goal}$ was tightened or loosened, PitG's conclusiveness rate varied wildly (dropping to 0% at $\omega_{goal} = 0.10$ ), whereas DPitG maintained a high conclusiveness rate (>97%) across the entire range.
Bias: The HDI+ROPE algorithm exhibited significant estimation bias ( $\hat{\theta}$ ) because it could stop on skewed, low-precision samples. Both PitG and DPitG maintained near-zero aggregate bias across all $\theta_{true}$ values. However, the paper notes that while the aggregate difference is negligible, some bias emerges as symmetry breaks (e.g., when $\theta_{true} \neq 0.5$ ), and within the ROPE, PitG shows slightly higher bias than DPitG for conclusive experiments, whereas the pattern reverses for inconclusive experiments.
Generalizability: The advantages of DPitG over PitG (higher conclusiveness with negligible extra cost) persisted as $\theta_{true}$ moved away from the null, and the framework extends to continuous outcomes and two-group comparisons.

4. Key Contributions

Proposed Algorithm (DPitG): A new sequential stopping rule that enforces a dual criterion (precision + decisiveness) to eliminate the high inconclusive rate of PitG while maintaining its near-zero bias properties.
Closed-Form Planning Formula: A derivation of $N_{goal} \propto V/\omega_{goal}^2$ , providing researchers with a pre-specified, pre-registration-compatible method to calculate the necessary sample size for a given precision goal.
Empirical Validation: Comprehensive simulation evidence showing that in the fair coin scenario, DPitG reduces the inconclusive rate of PitG from ~62% to ~2% at a median cost of only ~5% additional samples, without incurring the false-positive costs of coupled methods.
Practical Tools: The provision of an online interactive calculator and open-source code to facilitate the adoption of these methods in applied settings.

5. Significance and Claims

The paper positions DPitG as the method of choice for scenarios where a reliable verdict is required and the study can afford to collect at least $N_{goal}$ observations.

Resolution of Tension: DPitG resolves the inherent tension in PitG by making precision a necessary but not sufficient condition for stopping. It ensures that when a study stops, the result is both precise and decisive.
Scientific Rigor: The method supports the rigorous confirmation of the null hypothesis, a capability often lacking in NHST and currently compromised by high inconclusive rates in PitG. This is critical for fields like bioequivalence, "do-no-harm" testing, and futility stopping.
Pre-registration Compatibility: Because the stopping rule is fully pre-specified (based on $N_{goal}$ and $\omega_{goal}$ rather than data-dependent thresholds), it aligns with pre-registration standards and addresses design-stage imprecision implicated in the replication crisis.
Epistemic Honesty: The author argues that the precision requirement is a feature, not a flaw. It forces realistic planning and prevents analysts from declaring a verdict on insufficient data. If the data cannot support a conclusion within the budget, DPitG (like PitG) will not force a false conclusion, but unlike PitG, it minimizes the frequency of such "inconclusive" outcomes by extending the sampling budget only when necessary.

The paper concludes that while DPitG may require a modest increase in sample size compared to PitG in some scenarios, this cost is negligible compared to the gain in decisiveness and the avoidance of the systematic false-positive errors inherent in coupled methods like HDI+ROPE.