⚛️ high-energy experiments

Simulation Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data

This paper presents extensive simulation studies on the power of various goodness-of-fit and nonparametric two-sample tests for univariate data, demonstrating that no single method is universally effective and proposing a small, robust set of methods to ensure high power across diverse scenarios using the R packages R2sample and Rgof.

Original author: Wolfgang Rolke

Published 2026-11-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original author: Wolfgang Rolke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data, numbers rarely tell the whole story on their own. They arrive as collections of points, often representing measurements of height, temperature, or the time it takes for a machine to fail. Before a scientist can use these numbers to make predictions or draw conclusions, they must first ask a fundamental question: do these numbers come from the pattern we think they do? This is the realm of goodness-of-fit, a statistical check that compares a real set of observations against a theoretical model. Imagine a researcher who believes their data follows a bell-shaped curve, a common pattern in nature. They need a reliable way to confirm that belief or, just as importantly, to detect when the data has strayed into a different shape entirely. Sometimes, the challenge is not just checking one set of data, but comparing two separate groups to see if they were drawn from the same source. This is the two-sample problem, a task as old as the comparison of crops, medicines, or materials. For over a century, statisticians have developed various tools to answer these questions, but with so many methods available, a researcher faces a difficult choice: which tool is the right one for the job?

A researcher at the University of Puerto Rico set out to solve this dilemma by putting the most popular tools to the test. They did not rely on theory alone; instead, they built a massive digital laboratory where they could generate thousands of fake datasets with known properties. In this simulated environment, they created scenarios where the data differed from the expected pattern in very specific, subtle ways. Sometimes the difference was just a shift in the average value; other times, the data became more spread out, or skewed to one side, or developed a sharper peak. They tested these scenarios on both smooth, continuous data and on data that had been grouped into categories, which is common when dealing with large datasets or measurements from instruments with limited precision. The goal was simple but rigorous: to see which statistical method could most reliably spot the difference when it existed, without raising a false alarm when the data was actually as expected.

The results of this extensive simulation study revealed a surprising truth: there is no single "best" method that works perfectly in every situation. A test that excels at detecting a change in the average might fail completely when the data becomes more spread out. In fact, in many of the cases where the best method could detect a difference about 80% of the time, other methods failed to see it at all, succeeding less than 30% of the time. This inconsistency means that relying on a single favorite test is a gamble. However, the researcher found a practical solution. By selecting a small, specific group of methods for each type of problem, they could ensure that at least one tool in the group would perform nearly as well as the absolute best one for that specific scenario. In their simulations, this strategy worked so well that for every single case study they ran, at least one method in the recommended group was within 90% of the top performer's ability to detect the truth.

For data that flows smoothly without gaps, the study highlighted a set of tests based on likelihood ratios as the most powerful when the underlying model is fully known. However, the moment the researcher had to estimate unknown numbers from the data itself, the landscape changed. The top performers shifted to different methods, such as those that weigh the differences at the edges of the distribution more heavily. When the data was discrete, meaning it came in distinct chunks or bins, the winners were entirely different again. Here, tests that look at the smoothness of the data's distribution and those that compare the likelihood of the observed counts against the expected counts proved superior. The study also examined a popular alternative approach where researchers generate a second set of fake data to compare against the real data. While this hybrid method can be useful when the theoretical model is too complex to calculate directly, the simulations showed it consistently lagged behind the dedicated methods, even when the fake dataset was made five times larger than the real one.

The researcher packaged all their findings, along with the code to run these tests, into software tools available to the public. Their work serves as a guidebook for scientists navigating the complex terrain of data analysis. It confirms that while no single test is a universal key, a carefully chosen set of tools can unlock the truth in almost any situation. By following the specific recommendations for their type of data—whether continuous or discrete, and whether parameters are known or estimated—researchers can avoid the pitfall of using a method that is blind to the very differences they are trying to find. The study does not promise a magic bullet, but it offers a reliable map, ensuring that when scientists look for patterns in the noise, they are using the sharpest eyes available.

Technical Summary: Power Studies For Goodness-of-Fit and Two-Sample Methods For Univariate Data

Problem Statement
This study addresses the selection of statistical tests for two fundamental problems in univariate data analysis: the goodness-of-fit (gof) problem and the nonparametric two-sample problem. In the gof problem, the objective is to test $H_0: X \sim F$ against an unspecified alternative, where $F$ may be fully specified or contain unknown parameters estimated from the data. In the two-sample problem, the goal is to test $H_0: F = G$ to determine if two independent samples originate from the same distribution. The study encompasses both continuous and discrete data, the latter including histogram (binned) data, which is relevant for large datasets or situations with finite detector resolution (e.g., high-energy physics).

A central challenge identified is that no single test possesses uniformly high power across all possible alternatives. The author notes that a method performing optimally in one scenario may fail significantly in another, particularly when the null and alternative hypotheses agree on the first three moments (mean, variance, skewness) and differ only in the fourth (kurtosis) or vice versa.

Methodology
The author conducted a large-scale simulation study involving 120 goodness-of-fit case studies and 100 two-sample case studies. These cases were designed to be difficult, with alternatives constructed to differ from the null distribution in only one of the first four moments (mean, variance, skewness, or kurtosis) or through "mixed" differences involving multiple moments.

The study utilized the R packages Rgof and R2sample, which implement a comprehensive suite of methods for both continuous and discrete data. Key features of the implementation include:

Algorithmic Efficiency: Use of Rcpp and parallel programming to handle large datasets.
Data Types: Support for continuous, discrete, and weighted data, as well as random sample sizes (Poisson).
P-value Calculation: For the two-sample problem, p-values are derived via permutation methods or large-sample approximations. For goodness-of-fit, p-values are obtained via parametric bootstrap (simulation), particularly when parameters are estimated.
Hybrid Tests: The study evaluates "hybrid" approaches where a two-sample test is performed between the observed data and a Monte Carlo (MC) sample generated from the null hypothesis. Two MC sample sizes were tested: $n_{MC}=1$ (equal to data size) and $n_{MC}=5$ .

The performance of each method was evaluated based on statistical power at a nominal significance level of $\alpha = 0.05$ . The study also examined the stability of rankings across different sample sizes, significance levels, and target powers using leave-one-out cross-validation.

Key Contributions and Results
The primary contribution is the identification that no single method is universally superior. Instead, the author proposes small, problem-specific selections of methods such that, for every case study, at least one method in the selection achieves a power within 90% of the best-performing method.

1. Continuous Data:

Without Parameter Estimation: Zhang's tests ( $Z_C$ and $Z_A$ ) demonstrate the highest mean power in both goodness-of-fit and two-sample problems. They are within 90% of the best method in two-thirds of the two-sample case studies and 95% of variance-difference cases.
With Parameter Estimation: The advantage of Zhang's tests diminishes. Watson's test and the Anderson-Darling (AD) test emerge as having the highest mean power.
Recommended Selection: A combination of Zhang's $Z_C$ (or $Z_A$ ), Watson, Wasserstein, Anderson-Darling (or Cramér-von Mises), and a chi-square test with a small number of bins (10).

2. Discrete Data:

Goodness-of-Fit: Zhang's tests are not applicable (due to ties in ranks). The data-driven smooth tests (NS-B and NS-A) and the Berk-Jones (BJ) test show the highest mean power (71%, 68%, and 66%, respectively).
Two-Sample: The Epps-Singleton and HHG (Heller-Heller-Kaufman-Brill-Gorfine) tests are far superior, with mean powers of 83%.
Discretization Advantage: Surprisingly, the best method for discretized continuous data often outperforms the best method for the original continuous data, attributed to the strong performance of discrete versions of smooth, Berk-Jones, Epps-Singleton, and HHG tests.

3. Hybrid Methods:
Hybrid tests, which compare data to an MC sample, generally exhibit lower power than direct goodness-of-fit tests. While increasing the MC sample size ( $n_{MC}=5$ ) improves their performance, they do not surpass the power of standard gof tests. They are recommended only when the cumulative distribution function (CDF) under the null hypothesis cannot be calculated directly.

4. Cross-Validation and Stability:
Leave-one-out cross-validation confirms that the proposed selections are stable. The selection derived from all but one case study is identical to the full selection in at least 99% of cases. Using the recommended selection results in a mean power loss of less than 1 percentage point compared to the theoretical best method, whereas using a single "best" method (based on mean power) results in a loss of 5.7 to 12.3 percentage points.

Significance
The paper concludes that the choice of test must be tailored to the specific nature of the data (continuous vs. discrete) and the problem context (parameter estimation vs. none). The significance of this work lies in providing a rigorous, data-driven framework for selecting a small set of robust tests rather than relying on a single "best" method. The author emphasizes that while no single test is uniformly powerful, a carefully constructed selection ensures high power across a wide variety of difficult alternatives. The availability of the Rgof and R2sample packages allows researchers to apply these methods and compare new proposals against the established benchmarks presented in the study.

More like this