📊 statistics

Balancing Evidentiary Value and Sample Size of Adaptive Designs with Application to Animal Experiments

This paper proposes the Experimental Unit Information Index (EUII), a novel metric that balances statistical error rates and sample size to quantify the evidentiary value of individual experimental units, demonstrating its utility in optimizing adaptive animal experiments to reduce sample sizes while maintaining reliable inferences.

Original authors: Leonhard Held, Fadoua Balabdaoui, Saverio Fontana, Samuel Pawel

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonhard Held, Fadoua Balabdaoui, Saverio Fontana, Samuel Pawel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist running experiments on animals. Your goal is to find out if a new treatment works. But you also have a moral and practical duty: you want to use as few animals as possible (the "Reduce" principle) while still being absolutely sure your results are real and not just a lucky accident.

Usually, scientists play a tricky balancing act. If they use too few animals, they might miss a real effect. If they use too many, they are wasting resources. Traditionally, they compare different experiment designs by looking at two numbers: how often they make a false alarm (Type I error) and how often they successfully find a real effect (Power).

But this paper introduces a new way to look at the problem. It asks: "How much useful information does one single animal actually give us?"

Here is the breakdown of their idea using simple analogies:

1. The Problem with the Old Way

Imagine you are trying to guess if a coin is fair or biased.

Method A: You flip the coin 100 times. You get a clear answer.
Method B: You flip the coin 10 times, but you cheat a little bit by peeking at the results early and stopping if you see a pattern. You get an answer faster, but your "cheating" makes your error rate slightly higher.

Traditionally, comparing Method A and Method B is hard because they have different rules for "cheating" (error rates) and different costs (number of flips). You can't easily say which one is "better" just by looking at the raw numbers.

2. The New Tool: The "Animal Information Score" (EUII)

The authors created a new score called the Experimental Unit Information Index (EUII). Think of this as a "Value per Animal" score.

To understand this, they borrowed a concept from medical diagnostics called the Diagnostic Odds Ratio.

Imagine a medical test for a disease. A "good" test is one where a positive result strongly suggests you have the disease, and a negative result strongly suggests you are healthy.
The authors realized that a statistical experiment is just like a medical test. The "disease" is the hypothesis that the treatment works.
They calculated a score that tells you: "If I add one more animal to my experiment, how much does it improve my ability to tell the truth from a lie?"

If the score is high, every animal you use is working very hard to give you a clear answer. If the score is low, you are wasting animals on data that doesn't help much.

3. The "Early Stop" Trick (Adaptive Designs)

The paper focuses heavily on Adaptive Designs. This is like a detective who doesn't stick to a rigid plan.

The Plan: "I will interview 100 witnesses."
The Adaptive Twist: "I will interview witnesses one by one. If I find a smoking gun at witness #20, I stop immediately (Efficacy Stop). If I realize the story makes no sense at witness #30, I stop immediately because it's a waste of time (Futility Stop)."

This saves animals (or witnesses) because you don't interview the rest if you already know the answer. However, stopping early changes the math. It makes the "Value per Animal" calculation tricky because the number of animals used isn't fixed anymore.

4. What They Found

The authors took this new "Value per Animal" score and applied it to thousands of real past animal experiments (about 2,700 of them) to see what would have happened if they had used these "Early Stop" rules.

The Big Win: They found that using "Early Stop" rules (especially a specific type called the Pocock design) could save a massive number of animals. In their simulation, they could have saved over 13,000 animals across those studies by stopping early when the results were obvious.
The Trade-off: Sometimes, stopping early means you reject the "null hypothesis" (saying the treatment works) slightly less often than a rigid design. But the authors argue this is okay because the quality of the information per animal remaining is higher.
The Winner: Among the different "Early Stop" strategies, the Pocock method (which uses a consistent, slightly looser threshold for stopping early) generally gave the highest "Value per Animal" score. It was the most efficient at squeezing information out of every single animal used.

5. The "N-Hacking" Controversy

The paper also looked at a method called "N-hacking" (or "Constrained Sample Augmentation"), proposed by another researcher named Reinagel.

The Idea: If the results look "almost significant," add a few more animals to see if it crosses the line. If it looks hopeless, stop.
The Catch: This method technically increases the chance of a false alarm (Type I error).
The Verdict: The authors found that even with this higher error rate, Reinagel's method was very efficient at saving animals and had a high "Value per Animal" score. However, they noted that the Pocock method (which controls errors strictly) was still a very strong contender and often performed just as well or better depending on the situation.

The Bottom Line

This paper doesn't just say "use fewer animals." It provides a mathematical ruler to measure exactly how much information each animal contributes.

By using this ruler, they showed that stopping experiments early (when the answer is clear or when it's hopeless) is a smart way to follow the "Reduce" principle. It allows scientists to get reliable answers while using fewer animals, without having to sacrifice the reliability of the science. The Pocock design emerged as a top recommendation for achieving this balance in animal research.

Technical Summary: Balancing Evidentiary Value and Sample Size of Adaptive Designs with Application to Animal Experiments

Problem Statement
In animal research, the "Reduce" pillar of the 3R principles (Replace, Reduce, Refine) advocates for minimizing the number of experimental units while maintaining reliable statistical inferences. Standard statistical hypothesis testing typically focuses on controlling the Type-I error rate ( $\alpha$ ) and maximizing power, often ignoring the expected sample size or treating it as a fixed constraint. This creates a challenge when comparing different experimental designs, particularly adaptive designs (e.g., group-sequential designs) that allow for early stopping for efficacy or futility. Such designs result in random sample sizes, making direct comparisons of power and error rates difficult, especially when designs have different Type-I error rates or expected sample sizes. Existing performance scores often fail to account for sample size variations or require arbitrary weighting of components. The paper addresses the need for a unified metric that balances Type-I error, power, and sample size to quantify the evidentiary value of a single experimental unit.

Methodology
The authors propose a novel measure called the Experimental Unit Information Index (EUII). The methodology is built upon the following logical steps:

Diagnostic Likelihood Ratios: The authors adapt concepts from diagnostic testing to statistical hypothesis testing. They define positive and negative likelihood ratios ( $LR_+$ and $LR_-$ ) for a test result:
- $LR_+ = \text{Power} / \text{Type-I error rate}$
- $LR_- = (1 - \text{Power}) / (1 - \text{Type-I error rate})$
  These ratios relate the odds of the alternative hypothesis ( $H_1$ ) being true given a significant or non-significant result.
Diagnostic Odds Ratio (DOR): The DOR is defined as the ratio of these likelihood ratios ( $DOR = LR_+ / LR_-$ ). It quantifies the ability of a test to discriminate between $H_1$ and $H_0$ , interpretable both as the ratio of significance odds under $H_1$ versus $H_0$ (frequentist) and the ratio of posterior odds of $H_1$ given a significant versus non-significant result (Bayesian).
Normalization for Sample Size (EUII): To account for the number of experimental units ( $n$ ), the authors propose taking the $n$ -th root of the DOR:
$EUII = DOR^{1/n} = \frac{(LR_+)^{1/n}}{(LR_-)^{1/n}}$
This represents the geometric average contribution of a single experimental unit to the overall evidentiary value. A useful test has $EUII > 1$.
Extension to Adaptive Designs: For adaptive designs where the sample size $N$ is random, the authors generalize the EUII. Since the likelihood ratios act multiplicatively on odds, the index is defined using the expected inverse sample sizes for significant ( $N_+$ ) and non-significant ( $N_-$ ) findings:
$EUII = (LR_+)^{E(1/N_+)} / (LR_-)^{E(1/N_-)}$
This formulation accounts for the variability in sample size (via the coefficient of variation) and requires an input for the prior probability of the alternative hypothesis, $Pr(H_1)$ , to weight the expected inverse sample sizes under the null and alternative hypotheses.

Key Contributions and Results

Theoretical Properties: The paper derives the asymptotic value of the EUII for fixed sample sizes. For a one-sample $z$ -test, $\lim_{n \to \infty} EUII = \exp(\delta^2/2)$ , where $\delta$ is the standardized effect size. For a two-sample test, the limit is $\exp(\delta^2/8)$ . This demonstrates that the asymptotic evidentiary value depends solely on the assumed effect size under the alternative.
Motivating Case Study (Bonapersona et al., 2021): The authors re-analyzed 2,738 animal experiments in neuroscience and metabolism. By simulating interim analyses, they demonstrated that group-sequential designs with futility stopping could save substantial numbers of animals (e.g., the Pocock design with futility stopping saved ~13,500 animals in the dataset) while maintaining reasonable rejection rates.
Comparison of Group-Sequential Designs: In simulations comparing Pocock, O'Brien-Fleming, and Haybittle-Peto designs (with and without futility stopping), the EUII revealed that:
- Adaptive designs generally yield higher EUII values than fixed sample size designs.
- The Pocock method consistently demonstrated the highest EUII, particularly when combined with futility stopping. This is attributed to its constant nominal significance levels, which allow for earlier stopping under the null hypothesis, thereby reducing the expected sample size for non-significant findings.
- The EUII is sensitive to the prior probability $Pr(H_1)$ , though the relative ranking of methods often remains stable.
Evaluation of "Constrained Sample Augmentation" (N-hacking): The paper evaluated Reinagel's (2023) "constrained sample augmentation" method, which involves adding data if a "trend to significance" is observed, provided a futility stop is applied if $p > 0.1$ $p > 0.1$ .
- While this method inflates the Type-I error rate, the EUII analysis showed it can achieve high evidentiary value per unit, often outperforming standard group-sequential methods without futility stopping.
- However, when standard group-sequential methods (like Pocock) are also equipped with futility stopping (based on predictive power), they often match or exceed the EUII of Reinagel's approach, particularly for larger effect sizes.

Significance and Claims
The paper claims that the EUII provides a "measure of the evidentiary value of one experimental unit" that allows for the direct comparison of statistical designs with differing Type-I error rates, power levels, and expected sample sizes.

Practical Utility: The authors argue that the EUII is particularly useful for animal research where the goal is to maximize information per animal. It offers a single metric to balance the trade-offs between error rates and sample size efficiency.
Design Selection: The results suggest that for animal experiments, group-sequential designs with constant significance thresholds (like Pocock) combined with futility stopping are often optimal, as they minimize the expected sample size under the null hypothesis without sacrificing evidentiary value.
Modest Scope: The authors acknowledge that the EUII depends on the specification of the prior probability $Pr(H_1)$ and the assumed effect size. They note that in practice, EUII often requires estimation via simulation. The paper does not claim the EUII is a replacement for power calculations but rather a complementary metric for design selection and optimization, particularly in the context of the 3R principles.

The paper concludes that the EUII successfully integrates frequentist error rates and Bayesian posterior odds into a sample-size-adjusted metric, offering a robust framework for evaluating and optimizing adaptive experimental designs in preclinical research.