📊 statistics

E-Values For Multiplicity Control In Multiverse Analysis

This paper proposes using p-to-e calibration within generalized linear models to effectively control the false discovery rate in multiverse analyses, demonstrating that this approach significantly outperforms universal and soft-rank e-values in statistical power while successfully identifying associations between technology use and mental well-being issues in teenagers.

Original authors: Paul Rognon-Vael, David Rossell

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paul Rognon-Vael, David Rossell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for a single clue, you have a massive box containing thousands of potential clues: different types of evidence, various suspects, and countless ways to piece them together. In the world of science, this is called "multiverse analysis." It's a method where researchers don't just pick one way to look at their data; they try many different combinations of treatments, outcomes, and groups to see if a pattern holds up everywhere. Think of it like testing a new video game strategy on every possible map, with every character, and in every weather condition to see if it really works.

However, there's a tricky trap in this game. If you try enough combinations, you will eventually find a "winning" pattern just by pure luck, even if the strategy is actually terrible. This is the problem of "false positives." To stop researchers from getting fooled by luck, statisticians use tools to control the "False Discovery Rate" (FDR). Imagine FDR as a quality control inspector who makes sure that if you claim to have found a treasure, there's a very high chance it's actually gold and not just a shiny rock. One popular tool for this is the "p-value," which tells you how surprising your result is. But there's a newer, sturdier tool called the "e-value." You can think of an e-value like a betting score: if you bet $1 that your result is real, the e-value tells you how much your bet is worth. If the e-value is high, your bet is paying off big time; if it's low, you're likely just guessing.

This paper is about finding the best way to use these "e-value" betting scores when you are playing the massive "multiverse" game of testing many things at once. The authors, Paul Rognon-Vael and David Rossell, wanted to know: which method of calculating these e-values is the sharpest detective? They tested three different strategies to see which one could spot real effects without getting tricked by noise. They ran thousands of computer simulations—like playing the detective game millions of times in a virtual world—to see which method found the most true clues while ignoring the fake ones.

Here is what they discovered. They found that not all e-value calculators are created equal. Some methods, which they call "universal e-values" and "soft-rank e-values," were very safe but also very shy; they rarely raised the alarm, even when there was a real effect to find. It was like having a security guard who is so afraid of false alarms that they let actual thieves walk right past. However, a third method, which they call "p-to-e calibration," was much more effective. This approach takes the old-fashioned p-values and converts them into e-values using a specific mathematical recipe. In their simulations, this "calibrated" method was the champion: it found significantly more true effects than the other two methods while still keeping the false alarm rate under control.

The authors then took their best method and applied it to a real-world dataset about teenagers. They looked at how different types of technology use (like watching TV, playing video games, using the internet, or social media) were linked to mental well-being issues like depression, low self-esteem, and peer problems. In a previous study using a different method, researchers had concluded that technology use had almost no effect on teenagers' mental health. But when the authors used their new, sharper e-value tool, they found a very different story. They discovered strong, significant links between heavy internet and social media use and specific mental health struggles. For instance, they found that teenagers who used the internet heavily were much more likely to report low self-esteem and depressive symptoms, and their parents reported more peer problems. The odds of these issues were between 2 and 4 times higher for heavy users.

The paper suggests that while e-values are a powerful tool for keeping science honest in these complex "multiverse" scenarios, they work best when you have a lot of data. If the sample size is too small, even the best e-value method might miss the signal. But when the data is there, this "p-to-e calibration" method seems to be the most reliable way to separate the real gold from the shiny rocks, helping us understand the true impact of technology on our lives without getting lost in a sea of statistical noise.

Technical Summary: E-values for Multiplicity Control in Multiverse Analysis

Problem Statement
Multiverse analysis is an exploratory framework used to assess the association between multiple treatment definitions, multiple outcome definitions, and potentially multiple sub-populations. While valuable for detecting heterogeneity and sensitivity across specifications, the paper addresses the critical issue of using multiverse analysis for formal statistical inference. When researchers move beyond descriptive assessment to hypothesis testing, they face a massive multiple comparisons problem. Standard approaches, such as aggregating results into a single summary measure (e.g., Specification Curve Analysis), can be misleading when treatment effects are heterogeneous or when combining marginal and conditional associations. Conversely, testing each specification individually without correction leads to inflated false positive rates. The paper seeks a method to control the False Discovery Rate (FDR) in this setting that remains valid regardless of the dependence structure between tests and accounts for confounding covariates, without requiring the estimation of nuisance parameters often needed by empirical Bayes or q-value methods.

Methodology
The authors propose a framework based on e-values within a Generalized Linear Model (GLM) setting.

Statistical Framework: The analysis models $L$ outcomes and $p_x$ treatments across $p_z$ control covariates and $p_g$ sub-populations. The model estimates Average Treatment Effects (ATEs) for each treatment-outcome pair, allowing for sub-population-specific effects via interaction terms. The authors explicitly include all control covariates in the model to ensure strict Type I error control, contrasting with methods that average over subsets of covariates.
E-value Construction: The paper evaluates three strategies for generating e-values to test the null hypothesis that a specific treatment has zero effect:
- Universal Mixture E-values: Based on universal inference, utilizing a split-likelihood or a hybrid frequentist-Bayesian approach with Zellner's unit information prior to avoid arbitrary data splitting.
- Soft-rank E-values: Derived from permutation tests, where the test statistic is the likelihood ratio. These are robust to model misspecification (e.g., non-linear covariate effects) but computationally intensive.
- P-to-E Calibration: Converting standard p-values (from likelihood ratio tests) into e-values using specific calibrators, specifically a mixture calibrator ( $E_{c1}$ ) and an inverse root calibrator ( $E_{c2}$ ).
Multiplicity Control: The authors apply the e-BH procedure (and its refined/closed variants) to the set of $L \times p_x$ e-values. This procedure guarantees FDR control at level $\alpha$ regardless of the dependence structure among the tests.

Key Contributions and Simulation Results
The paper contributes a comparative analysis of these e-value strategies through simulation studies in linear and logistic regression settings, mimicking the dimensions of social science multiverse analyses.

Type I Error Control: All proposed e-value strategies successfully controlled the Type I error rate below the specified significance levels ( $\alpha = 0.01, 0.05$ ), often behaving conservatively.
Statistical Power: The simulations revealed significant differences in power.
- P-to-E Calibration ( $E_{c1}$ and $E_{c2}$ ): These methods significantly outperformed universal mixture and soft-rank e-values, particularly for moderate sample sizes and effect sizes. The mixture calibrator ( $E_{c1}$ ) was found to be the most powerful, accumulating stronger evidence against the null for truly alternative hypotheses.
- Universal Mixture ( $E_{um}$ ): This approach was found to be extremely conservative, often failing to detect effects unless sample sizes or effect sizes were very large.
- Soft-rank ( $E_{sr}$ ): While less powerful than calibrated e-values for small $n$ , they became competitive for larger sample sizes and offer robustness against model misspecification.
Practical Application: The methodology was applied to the Millennium Cohort Study (MCS) to investigate the association between teenager technology use and mental well-being.
- Previous analyses using Specification Curve Analysis concluded the association was negligible.
- The proposed e-value framework, controlling FDR at 0.05, rejected four specific hypotheses, identifying significant associations between high internet/social media usage and depressive symptoms, low self-esteem, and peer problems. The estimated odds ratios ranged from 2 to 4, indicating practical relevance alongside statistical significance.

Significance and Claims
The paper positions e-values as a "conservative procedure" that is a useful complement to multiverse analysis when hypothesis testing is required.

Validity: The primary advantage highlighted is that e-values provide finite-sample FDR control that does not rely on assumptions of independence or positive dependence between tests, nor do they require estimating nuisance parameters (like the proportion of true nulls).
Limitations: The authors are modest regarding the statistical power of e-values. They acknowledge that, compared to methods with stronger assumptions (like q-values or empirical Bayes), e-values may yield fewer significant findings, particularly in small samples.
Recommendation: The authors recommend reporting treatment effects for all specifications separately and applying multiplicity control, rather than relying on aggregate summary measures which can obscure heterogeneous effects. They specifically recommend the mixture p-to-e calibration ( $E_{c1}$ ) for its superior power in typical multiverse settings, provided the underlying model is approximately correct. If model misspecification is a major concern, soft-rank e-values are suggested as a robust alternative, albeit with a power cost.

In conclusion, the paper argues that while e-values may limit the number of discoveries due to their conservative nature, they offer a rigorous, assumption-light framework for valid statistical inference in the complex, high-dimensional landscape of multiverse analysis.

Technical Summary: E-values for Multiplicity Control in Multiverse Analysis

More like this