🧬 biology

Equivalence testing in pesticide risk assessment -- Evaluation and practical guidance for design, analysis and interpretation

This paper evaluates equivalence testing frameworks for pesticide risk assessment, demonstrating that while increased site replication is necessary for reliable regulatory decisions, the EFSA's original approach outperforms alternative methods in detecting harmful effects, and that statistical power can be maintained with fewer sites through covariate adjustment or anticlustering randomization.

Original authors: Dimitry Wintermantel, Julia Osterman, Magdalena M. Mair, Florian Hartig

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dimitry Wintermantel, Julia Osterman, Magdalena M. Mair, Florian Hartig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a judge in a courtroom, but instead of deciding if someone is guilty of a crime, you are deciding if a new pesticide is safe for honeybees. For a long time, the rule of the game was: "If we can't prove the pesticide is hurting the bees, then it must be safe."

The problem with this old rule is that sometimes the courtroom is too small, the witnesses are too confused, or the evidence is too messy to see the harm even when it's there. It's like trying to hear a whisper in a noisy stadium; just because you didn't hear it doesn't mean the whisper wasn't there. This paper argues that this "silence means safety" approach has let dangerous pesticides slip through the cracks.

Here is a breakdown of the paper's findings using simple analogies:

1. The New Rule: "Prove You Are Safe, Not Just Harmless"

The European Food Safety Authority (EFSA) has changed the rules. Now, the pesticide company (the defendant) must prove that their product causes less than a 10% drop in the size of a bee colony. This is called an Equivalence Test.

The Old Way (Difference Test): "Did the bees die? If we can't prove they died, you win." (This is like saying, "If I can't prove you stole my wallet, you didn't steal it," even if I didn't look very hard).
The New Way (Equivalence Test): "Show us that the damage is so small it's below the 10% line. If you can't prove it's below that line, you lose."

2. The Controversy: Two Different Rulers

Some critics (Hotopp et al.) argued that the new rule is too strict. They said, "Bees naturally have ups and downs in their population, like a rollercoaster. If we compare the pesticide bees to the average of the control bees, we might reject safe pesticides just because of natural noise."

They proposed a new "ruler": Instead of comparing the pesticide to the average of the safe bees, compare it to the lowest point of the safe bees' confidence range. They argued this would make it easier to pass the test.

The Paper's Verdict:
The authors ran thousands of computer simulations (like running a video game a million times to see what happens) and found that the critics' new ruler is actually a loose ruler.

The Analogy: Imagine the safety limit is a height requirement for a rollercoaster (you must be under 6 feet to be safe).
- The EFSA rule says: "Measure the person's height. If they are under 6 feet, they are safe."
- The Critics' rule says: "Measure the person's height, but then subtract a little bit of 'wiggle room' before measuring. If they are under 6 feet after subtracting wiggle room, they are safe."
The Result: The critics' rule lets in people who are actually too tall (dangerous pesticides) because the "wiggle room" hides their true height. The paper shows that the EFSA rule is the only one that reliably keeps the dangerous ones out.

3. The Cost of Safety: How Many Beehives Do We Need?

To make sure the test is accurate (statistically "powered"), you need enough data. The paper found that to be 100% sure a pesticide is safe, you need to test it in many different locations (sites) with many beehives.

The Problem: Current studies often use too few sites. It's like trying to judge the quality of a whole forest by looking at just one tree.
The Solution: The paper shows that if you have a pesticide that is truly safe (or causes very little harm, like a 5% drop), you don't need as many sites as the old rules suggested, BUT you do need more than we currently use.

4. The Magic Trick: "Balancing the Scales"

The paper offers a clever way to reduce the number of beehives needed without losing accuracy. It's called Anticlustering Randomization.

The Analogy: Imagine you are dividing a group of people into two teams for a race. If you just pick names out of a hat, one team might accidentally get all the tall, fast runners, and the other gets the short, slow ones. The race results would be unfair, not because of the shoes they wore (the pesticide), but because of their height.
The Fix: Before the race starts, you carefully pair the tall runners with the short runners and split them evenly between teams. This ensures both teams start on equal footing.
In the Paper: The researchers created a computer tool (an R function) that helps scientists arrange their beehives so that strong colonies and weak colonies are perfectly balanced between the "safe" group and the "pesticide" group. This "leveling of the playing field" means you need fewer beehives to get a clear answer, saving money and time while keeping the bees safe.

5. The Final Takeaway

Don't trust the "loose ruler": The alternative test proposed by critics makes it too easy to approve dangerous pesticides.
Do trust the EFSA rule: It is strict, but it is the only one that reliably protects bees.
Do the math right: We need more field sites than we currently use, but we can cut down the number needed by using smart planning (balancing the bees) before the experiment starts.
Interpretation matters: If a study fails to prove a pesticide is safe, it doesn't automatically mean the pesticide is dangerous; it just means we haven't proven it's safe yet. We shouldn't call it "safe" just because we didn't find proof of harm.

In short: To protect our pollinators, we need to stop guessing and start measuring with a strict, fair ruler, while using smart organization to make sure our measurements are clear.

Technical Summary: Equivalence Testing in Pesticide Risk Assessment

Problem Statement

Regulatory risk assessments for pesticides, particularly for honeybees (Apis mellifera), have historically relied on difference testing (significance testing against a null hypothesis of "no effect"). This approach is prone to Type II errors (false negatives) in underpowered studies, where large detrimental effects may go undetected due to high variance or low replication. Consequently, pesticides harmful to pollinators have occasionally passed regulatory approval only to be restricted or banned later.

While the European Food Safety Authority (EFSA) revised its 2023 guidance to shift toward equivalence testing (specifically, a non-inferiority test) to ensure that pesticide effects do not exceed a Specific Protection Goal (SPG) of a 10% colony size reduction, this approach has faced criticism. Critics, notably Hotopp et al. (2024), argue that the EFSA framework requires excessive site replication, making regulatory approval impractical. They proposed an alternative "Hotopp test," which defines treatment effects relative to the lower bound of the 90% confidence interval of the control group, arguing this accounts for natural variability. The core problem addressed by this paper is whether the EFSA recommendation or the Hotopp alternative better protects against falsely classifying high-risk pesticides as "low risk," and what the actual sample size requirements are for reliable assessment.

Methodology

The authors conducted extensive parametric bootstrap simulations mimicking regulatory honeybee field studies, using control data from a real-world industry study (Rolke et al., 2016) as the baseline for distributional parameters.

Simulation Design:

Data Generation: The authors simulated datasets with varying numbers of sites (4 to 45) and fixed colonies per site (6). They generated initial bee counts and subsequent time-point data using a Generalized Linear Mixed Model (GLMM) with a negative binomial distribution, incorporating random effects for site and colony.
Treatment Effects: Simulated pesticides were assigned true effect sizes ranging from 0% to -20% (reduction in colony size).
Adjustment Strategies: The study evaluated four strategies to handle initial colony size variability:
1. No adjustment: Ignoring initial size.
2. Design adjustment: Using a new R function (experimental_allocation()) to perform anticlustering randomisation, balancing initial bee numbers across treatments and sites before the experiment.
3. Statistical adjustment: Including initial bee size as a covariate in the GLMM.
4. Combined adjustment: Utilizing both design and statistical adjustments.
Test Comparison: The simulations compared three statistical approaches:
1. EFSA Equivalence Test: A one-sided non-inferiority test where the null hypothesis is that the effect exceeds the 10% SPG (tested at $\alpha = 0.2$ ).
2. Hotopp Equivalence Test: An alternative test shifting the baseline to the lower bound of the control's 90% CI.
3. Difference Test: Traditional significance testing (one-sided, $\alpha = 0.05$ ).
Evaluation Metrics: The authors analyzed the trade-off between false "low-risk" classifications (Type I error in the context of equivalence, or false trust) and false "high-risk" classifications (Type II error, or false mistrust) using Pareto plots. They also determined the number of sites required to achieve 80% power for classifying pesticides with true effect sizes of 0% and -5% as "low risk."

Key Results

1. Comparison of EFSA vs. Hotopp Tests

Statistical Trade-offs: The EFSA and Hotopp tests were found to be statistically equivalent in their ability to distinguish high-risk from low-risk pesticides if their $\alpha$ levels were adjusted to align protection rates. However, the Hotopp test is calibrated differently: at a nominal $\alpha = 0.2$ , it effectively operates with a much higher error rate, leading to significantly higher rates of falsely classifying high-risk pesticides as "low risk."
False Trust Rates: Under the recommended sample size (10 sites, 6 colonies), the Hotopp test misclassified pesticides with a true -12% effect as "low risk" in 50% of cases and -15% effects in 33% of cases. In contrast, the EFSA test maintained false trust rates below the nominal 20% threshold for all tested effect sizes.
Conclusion on Hotopp Test: The authors conclude that the Hotopp test does not offer a statistical advantage but rather lowers the stringency of the test, potentially incentivizing increased background variance to pass equivalence.

2. Sample Size Requirements

Feasibility for Low-Risk Pesticides: For pesticides with a true effect size of 0%, the EFSA equivalence test is feasible with current or slightly increased replication (6–7 sites with adjustment; 9 sites without).
Requirements for Moderate Effects: For pesticides with a true effect size of -5% (a tolerable but negative effect), the study found that 24–25 sites are required with adjustment strategies, compared to ~34 sites without.
Comparison to Previous Guidance: The site requirements for the new EFSA guidance (equivalence test) are generally lower than those implied by the 2013 guidance (difference test with a 7% threshold), which required 29–41 sites to detect a 7% reduction with 80% power.

3. Impact of Covariate Adjustment

Variance Reduction: Both anticlustering randomisation (design adjustment) and statistical adjustment (covariate inclusion) substantially reduced the number of sites required to achieve 80% power.
Efficiency: Adjusting for initial colony size reduced the site requirement for a -5% effect size by approximately 10 sites per treatment compared to unadjusted designs. The authors note that design adjustment is preferable as it does not reduce degrees of freedom and is immune to post-hoc analysis manipulation.

Key Contributions

Validation of EFSA Guidance: The paper provides simulation-based evidence that the EFSA 2023 equivalence testing framework is statistically robust and effectively controls the rate of false "low-risk" conclusions at the specified $\alpha$ level, unlike the proposed Hotopp alternative.
Practical Sample Size Estimates: The study quantifies the specific number of sites needed to reliably demonstrate safety for pesticides with varying effect sizes, clarifying that while increased replication is necessary for moderate effects, it is often lower than previously assumed under the old difference-testing paradigm.
Methodological Tools: The authors introduce and validate anticlustering randomisation via a new R function (experimental_allocation()) as a cost-effective design strategy to reduce variance and site requirements without compromising statistical power.
Interpretation Framework: The paper proposes a nuanced interpretation of equivalence test results, distinguishing between "demonstrated low risk," "demonstrated high risk," and "high risk not excluded" (non-significant results), arguing against the binary interpretation of non-significant equivalence tests as definitive proof of high risk.

Significance and Claims

The paper claims that the shift to equivalence testing is a necessary and scientifically sound evolution for regulatory risk assessment. It argues that:

Protection Goals are Met: The EFSA equivalence test ensures that the Specific Protection Goal (SPG) is met with a controlled probability of error, placing the burden of proof on the chemical industry to demonstrate safety with sufficient power.
No Free Lunch: Attempts to lower the stringency of the test (as proposed by Hotopp et al.) to reduce sample size requirements inevitably increase the risk of approving harmful pesticides.
Feasibility: With the implementation of variance-reduction strategies like balanced allocation, the sample size requirements for the new guidance are feasible and often lower than those required by the previous guidance for detecting smaller effect sizes.
Generalizability: These principles extend beyond honeybees to any higher-tier ecotoxicity study, advocating for clear protection goals, non-inferiority testing, and pre-experimental variance reduction strategies.

The authors conclude that broad adoption of these practices—specifically the EFSA equivalence test combined with balanced allocation—will lower the probability of authorizing harmful pesticides and align regulatory outcomes with environmental protection goals.