📊 statistics

Weighted Extensions of the Kolmogorov-Smirnov, Cramer-von Mises, and Anderson-Darling Tests for Assessing Covariate Balance

This paper extends the Kolmogorov-Smirnov, Cramer-von Mises, and Anderson-Darling tests to accommodate weighted data for covariate balance assessment in causal inference, demonstrating through simulations that these weighted versions maintain valid Type I error control and preserve their respective power advantages against different discrepancy types, with the Anderson-Darling test recommended as a robust general-purpose default.

Original authors: Ariel Linden

Published 2026-07-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ariel Linden

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Did a new medicine actually work, or did the patients just happen to be healthier to begin with? In the world of science, this is called "causal inference." To be sure, you need to compare two groups of people: those who got the treatment and those who didn't. If the groups are perfectly matched—like twins in height, weight, and age—any difference in their health later can be blamed on the medicine. But in real life, we can't always run perfect experiments where people are randomly assigned. Instead, scientists often use a clever trick called "weighting." Think of it like a digital scale in a grocery store. If the "treated" group has too many heavy people and the "untreated" group has too many light people, the scientist assigns a "weight" to each person. A light person in the heavy group gets a big weight (like a heavy counterweight) to balance the scale, while a heavy person in the light group gets a tiny weight. This mathematically forces the two groups to look like they are balanced.

But here is the tricky part: just because the average weight looks the same doesn't mean the groups are truly identical. Imagine two bags of marbles. One bag has ten 1-pound marbles. The other has five 1-pound marbles and five 2-pound marbles. If you just look at the average, they might seem similar, but the distribution of weights is totally different. Scientists need to check if the entire shape of the data matches, not just the average. For years, they had tools to check this, but those tools broke when you tried to use them on these "weighted" bags of marbles. They couldn't handle the digital counterweights. This paper is about fixing those tools so they work perfectly, even when the data has been tweaked with weights.

The paper introduces three new, upgraded versions of classic "balance tests" that can handle these weighted data. The authors, led by Ariel Linden, took three famous statistical methods—the Kolmogorov–Smirnov (KS), the Anderson–Darling (AD), and the Cramér–von Mises (CVM)—and taught them how to read the weights. They didn't just guess how to do it; they built a new, universal "shuffling" method. Imagine you have a deck of cards where some cards have heavy stickers on them. Instead of trying to figure out why the stickers are there, the new method simply shuffles the labels (who is in which group) while keeping the cards and their stickers exactly where they are. By doing this thousands of times, they can see if the differences they see are real or just a fluke.

The researchers tested these new tools in a massive computer simulation. They created four different scenarios to see how the tools performed. First, they checked if the tools made false alarms (Type I error) when the groups were actually identical. The results were excellent: all three tools stayed very close to the correct 5% error rate, meaning they didn't cry wolf when there was no wolf.

Then, they tested the tools against three different types of "mismatches" between groups:

The Central Mismatch: Imagine the groups are identical everywhere except right in the middle, like two crowds where everyone in the middle is wearing red shirts in one group and blue in the other. Here, the Kolmogorov–Smirnov (KS) test was the clear winner. It is like a detective who only looks for the single biggest gap; it found the middle mismatch faster than anyone else.
The Tail Mismatch: Imagine the groups are identical in the middle, but the "extreme" people (the very tall or very short) are different. This is like one group having a few giants and the other having a few dwarfs. Here, the Anderson–Darling (AD) test was overwhelmingly the best. It is like a detective who pays extra attention to the edges of the room. The KS test barely noticed this problem at all.
The Diffuse Mismatch: Imagine the groups are slightly different everywhere, like one group is just generally a bit taller than the other across the board. In this case, both the AD and CVM tests performed very well and were almost tied, while the KS test lagged behind.

The paper also looked at what happens when the "weights" themselves are very messy and vary a lot (simulating real-world data where some people get huge weights and others tiny ones). Even with this chaos, the tools held up. The AD test remained the best at spotting tail problems, and the KS test remained the best for central problems.

So, what's the takeaway for a scientist? If you suspect the groups are different right in the middle, use the KS test. If you are worried about the extremes (the tails), the AD test is your best friend. If you aren't sure where the difference might be, or if you think the difference is spread out everywhere, the AD test is a safe, reliable "default" choice that works well in almost every situation, often beating the others. The authors even built these tools into software commands (for a program called Stata) so other researchers can use them right away.

In short, this paper didn't just invent new tools; it fixed old, broken ones so they can handle the messy, weighted data that real-world science produces. It confirmed that while no single tool is perfect for every job, the Anderson–Darling test is the most versatile "all-rounder" for checking if two groups are truly balanced, especially when you are worried about the extremes.

Technical Summary: Weighted Extensions of Distributional Goodness-of-Fit Tests for Covariate Balance

Problem Statement
Assessing covariate balance is a fundamental diagnostic step in causal inference, required to support the assumption that unmeasured characteristics do not meaningfully confound estimated treatment effects. While balance is often achieved through post-hoc adjustments like propensity-score weighting, entropy balancing, or stratification, current diagnostic practices rely heavily on summary measures such as the standardized mean difference (SMD) and variance ratios. These summary statistics possess a critical limitation: they target only specific points in a covariate's distribution (e.g., the mean or variance) and are inherently unable to detect distributional differences in skewness, mass location, or tail behavior that do not affect those specific moments.

Distributional goodness-of-fit tests based on the empirical cumulative distribution function (ECDF)—specifically the Kolmogorov–Smirnov (KS), Cramér–von Mises (CVM), and Anderson–Darling (AD) tests—offer a more comprehensive alternative by comparing entire distributions. However, until this work, these tests were available only for unweighted data. This gap prevented their application in the very settings where distributional balance is most critical: weighted analyses intended to equate treated and comparison groups.

Methodology
The paper extends the KS, CVM, and AD tests to accommodate case weights of any origin. The core methodological innovation is a unified label-permutation inference procedure that treats an observation's weight as a fixed attribute rather than a quantity tied to a specific estimation mechanism. This approach requires no assumptions regarding how the weights were generated, allowing the tests to be applied to any weighting scheme (e.g., inverse probability of treatment weighting, entropy balancing) without modification.

The weighted statistics are defined as follows:

Weighted KS ( $D^w$ ): The maximum absolute vertical distance between the weighted ECDFs of the two groups.
Weighted CVM ( $CVM^w$ ): The sum of squared discrepancies between the weighted ECDFs across distinct values, without variance standardization.
Weighted AD ( $AD^w$ ): The sum of squared discrepancies standardized by the variance under the null. Crucially, the sample size $N$ in the variance term is replaced by the Kish effective sample size ( $n_e$ ) to account for the information content of the weighted sample. The tie multiplier remains the raw count of observations sharing a value.

Inference is conducted via label permutation: group labels are randomly reshuffled among $N$ observations while keeping each observation's value and weight fixed. The $p$ -value is derived from the proportion of permuted statistics exceeding the observed statistic.

Simulation Study Design
A simulation study was conducted to evaluate whether the known comparative advantages of the unweighted tests persist under weighting. Four scenarios were constructed:

Null: No discrepancy (to assess Type I error).
Centrally Located Discrepancy: A shift confined to the interquartile range.
Tail-Located Discrepancy: A shift confined to the extreme tails.
Diffuse Discrepancy: A uniform shift across the entire distribution.

The study varied sample sizes ( $n = 1,000$ to $4,000$) and weight variability (target Kish effective sample size ratio $r_e = 0.6$ , with sensitivity checks from $0.2$ to $0.9$).

Key Results

Type I Error Control: Across all sample sizes and under substantial weight variability, all three weighted tests controlled Type I error close to the nominal 5% level. This confirms that the shared label-permutation procedure effectively calibrates the null distribution without requiring assumptions about weight generation.
Centrally Located Discrepancy: The KS test demonstrated the highest power, significantly outperforming both AD and CVM. This aligns with the theoretical expectation that KS is most sensitive to a single, large localized divergence.
Tail-Located Discrepancy: The AD test was overwhelmingly the most powerful, significantly outperforming both KS and CVM. KS and CVM remained close to their Type I error rates, failing to detect the tail discrepancy. This confirms AD's sensitivity to tail deviations due to its variance-standardization term.
Diffuse Discrepancy: Both AD and CVM performed comparably and significantly outperformed KS. Contrary to the theoretical expectation that CVM would be uniquely superior to diffuse discrepancies, AD held a modest but statistically significant advantage over CVM at moderate to large sample sizes.
Weight Variability Sensitivity: The comparative ordering of the tests remained stable across a wide range of weight variability ( $r_e$ from 0.2 to 0.9). At the most extreme variability ( $r_e = 0.2$ ), all three tests showed a modest upward deviation in Type I error (approx. 6.4–6.7%), though the relative performance rankings remained consistent.

Applied Example
The methods were applied to a quasi-experimental evaluation of a health coaching intervention using inverse-probability-of-treatment weights. Assessment of Body Mass Index (BMI) balance revealed divergent conclusions: the AD test detected a significant residual discrepancy ( $p = .018$ ), while CVM was suggestive ( $p = .079$ ) and KS was non-significant ( $p = .123$ ). Visual inspection of the weighted ECDFs showed a gradual, persistent divergence across a range of BMI values rather than a single sharp gap, illustrating why AD (sensitive to whole-distribution shifts) detected the imbalance while KS did not.

Significance and Claims
The paper makes two primary contributions:

Methodological Extension: It provides the first weighted generalizations of the KS, CVM, and AD tests, making them available for routine covariate balance assessment in weighted causal inference studies.
Inference Framework: It establishes a single, assumption-free label-permutation inference procedure applicable to any weighting scheme.

The authors conclude that the known unweighted comparative advantages of these tests are preserved under weighting. Consequently, they recommend Anderson–Darling as a reasonable general-purpose default for routine covariate balance assessment, as it performs robustly across discrepancy types and is overwhelmingly superior for tail-located discrepancies (a common concern in propensity-score weighting). However, they note that Kolmogorov–Smirnov retains a distinct advantage if a centrally concentrated imbalance is specifically suspected. The paper suggests that reporting multiple tests (e.g., both AD and KS) is a prudent strategy to guard against the blind spots of any single statistic.

The authors remain modest regarding the scope of their findings, noting that the simulations used a normal base distribution and that the comparative ordering for diffuse discrepancies (where AD and CVM were comparable) warrants further investigation into the underlying mechanisms. They do not claim to have proven the validity of these tests under every possible weighting scheme or distributional shape, but rather demonstrate consistency under the specific conditions simulated.

Technical Summary: Weighted Extensions of Distributional Goodness-of-Fit Tests for Covariate Balance

More like this