📊 statistics

Why Empirical p-Values Are Not Uniform: Reference Samples, Dependence, and PIT Backtesting

This paper demonstrates that empirical p-values derived from finite reference samples deviate from the expected uniform distribution due to induced dependence and variance distortions, necessitating revised backtesting calibration methods that account for the underlying two-stage sampling structure rather than treating them as independent uniform draws.

Original authors: Jakub Lis

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jakub Lis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Perfect Map" vs. The "Rough Sketch"

Imagine you are trying to judge how "average" a new person is compared to a group of people you already know.

In the perfect world of statistics (the Theory), you have a "God's eye view." You know the exact, perfect map of how everyone in the world is distributed. If you take a new person and ask, "What percentage of people are shorter than you?" the answer is a Probability Integral Transform (PIT). If your model is correct, these answers should be perfectly random numbers between 0 and 1, like drawing a number out of a hat.

The Problem: In the real world (the Practice), we don't have the perfect map. We only have a Reference Sample—a small group of people we've measured so far. To answer the question "What percentage are shorter?", we have to guess based on our small group. We create a "rough sketch" of the distribution using this limited data.

The Paper's Discovery: The author, Jakub Lis, points out a hidden trap. When we use this "rough sketch" (the empirical estimate) instead of the "perfect map," the numbers we get stop behaving like random numbers. They become distorted. If we treat them as if they were perfect random numbers, our statistical tests will give us the wrong answers.

The Three Scenarios: How We Build the Sketch

The paper looks at three different ways people try to build this "rough sketch" to test their models. Each method breaks the rules in a different way.

1. The "One Fixed Group" Method (Common-Sample)

The Analogy: Imagine you are a teacher grading 100 students. To decide if a student is "average," you compare them to one single, fixed class of 50 students you measured at the start of the year. You use that same class of 50 to grade every single one of the 100 new students.

What Goes Wrong:
Because you are using the same 50 students for everyone, any mistake or weirdness in that specific group of 50 gets copied onto all 100 new grades.

If that group of 50 happened to be unusually tall, your "average" line shifts up for everyone.
The paper shows that the math here stops looking like a "One-Sample Test" (checking one group against a perfect rule) and starts looking like a "Two-Sample Test" (comparing two messy groups against each other).
The Result: If you use standard tests that assume the numbers are independent, you will get false alarms. You might think your model is broken when it's actually fine, or vice versa.

2. The "Fresh Group for Everyone" Method (Independent Reference)

The Analogy: Now, imagine that for each of the 100 students, you pull out a brand new, different group of 50 students to compare them against.

What Goes Right:
This is the only method that works well. Because the "noise" or mistakes in the reference groups are different for every student, they cancel each other out.

One group might be too tall, the next too short. When you average them all out, the errors disappear.
The Result: The numbers behave almost exactly as they should. The standard tests work here because the "rough sketches" average out to look like the "perfect map."

3. The "Sliding Window" Method (Rolling Window)

The Analogy: This is like a sliding door. You compare a student to the 50 people who walked through the door just before them. Then, that student walks through the door and becomes part of the group for the next person.

What Goes Wrong:
This creates a chain reaction.

Student A helps define the group for Student B.
Student B (who was influenced by A) helps define the group for Student C.
The numbers are no longer independent; they are "sticky" or autocorrelated.
The Result: The paper finds that this method suppresses the natural "wobble" or variance in the data. The numbers look too stable, too perfect. If you run a standard test on this, you might miss real problems because the data looks deceptively smooth.

The Core Takeaway

The paper argues that we have been treating these "rough sketches" (empirical p-values) as if they were "perfect maps."

The Mistake: We assume that because the theory says the numbers should be uniform (random), the numbers we calculate from real data are also uniform.
The Reality: The act of estimating the map using a finite sample changes the nature of the numbers.
- If you use one fixed group, you are actually doing a two-group comparison, not a one-group check.
- If you use a sliding window, your numbers are linked together like a chain, breaking the rule of independence.

The Conclusion:
To fix this, we cannot just use the standard "goodness-of-fit" tests (like the Kolmogorov–Smirnov test) that assume the data is independent and perfectly uniform. We need new, revised methods that account for the fact that we are building our map out of a limited supply of bricks. If we don't adjust for this, our backtesting (checking if our risk models work) could be misleading.

Technical Summary: Why Empirical p-Values Are Not Uniform

Problem Statement
The paper addresses a fundamental discrepancy between the theoretical properties of the Probability Integral Transform (PIT) and its practical implementation in model validation and backtesting. Theoretically, if $X$ is a continuous random variable with cumulative distribution function (CDF) $F_X$ , the transformed variable $F_X(X)$ follows a uniform distribution on $[0, 1]$ . This property underpins the use of PITs and empirical p-values to assess the calibration of predictive distributions, a practice widely adopted in fields ranging from ensemble forecasting to financial risk management (e.g., European Central Bank guidelines).

However, in practice, the true distribution $F_X$ is unknown. Consequently, practitioners must estimate percentiles using finite reference samples. The paper argues that this estimation step fundamentally alters the statistical structure of the problem. Specifically, treating empirical percentiles as independent draws from a uniform distribution is incorrect; the resulting statistics do not follow the expected uniform distribution even under correct model specification, thereby invalidating classical one-sample goodness-of-fit tests (such as the Kolmogorov–Smirnov test) when applied directly to these empirical values.

Methodology
The author employs a combination of theoretical decomposition and Monte Carlo simulations to analyze the distributional properties of empirical p-values under three distinct implementation frameworks:

Single Conditioning Sample (Fixed Reference): All observations are evaluated against a single, fixed reference sample of size $n$ . The empirical percentile $\hat{p}(x)$ is constructed via interpolation of the order statistics of this reference sample.
Independent Reference Samples: Each observation is evaluated against a distinct, independent reference sample. This framework mimics the "rank histogram" approach used in ensemble forecasting.
Rolling Window Framework: Empirical p-values are computed using a moving window of preceding observations, where each new observation enters the reference set for subsequent evaluations.

The analysis relies on decomposing the empirical distribution function of the transformed values, $F_m$ , relative to the uniform distribution $F_U$ . The paper utilizes Donsker's theorem to characterize the asymptotic behavior of empirical processes and compares the resulting Kolmogorov–Smirnov (KS) statistics and p-values against their theoretical counterparts.

Key Contributions and Results

Decomposition of Variance (Fixed Reference):
The paper demonstrates that when a single reference sample is used, the empirical distribution of the transformed values is a superposition of two sources of randomness:
1. Sampling variability from the evaluated observations (order $O_p(m^{-1/2})$ ).
2. Estimation error from the finite reference sample (order $O_p(n^{-1/2})$ ).
The decomposition is expressed as $F_m - F_U = (F_m - \tilde{F}) + (\tilde{F} - F_U)$ , where $\tilde{F}$ is the random distribution induced by the reference sample. The results show that the resulting KS statistic behaves asymptotically like a two-sample KS statistic rather than a one-sample statistic. Numerical experiments confirm that the dispersion of empirical p-values and the resulting KS statistics are nearly indistinguishable from those generated by comparing two independent samples. Consequently, applying standard one-sample uniformity tests leads to distorted inference and systematic rejection rates.
Mitigation via Independent Samples:
In the framework where each observation has its own independent reference sample, the estimation noise associated with the reference samples averages out across observations. The paper shows that in this setting, the dispersion of empirical percentiles closely tracks that of exact percentiles. While noise is present, the transformed values retain the statistical properties of the underlying sample, and the correlation between exact and empirical KS statistics remains evident.
Dependence in Rolling Windows:
The rolling-window implementation introduces autocorrelation and variance suppression. Because each observation determines its own p-value and subsequently enters the reference set for future evaluations, the sequence of empirical p-values is no longer independent. Simulations indicate that when the number of evaluated observations ( $m$ ) is comparable to the window size ( $n$ ), the correlation between the KS statistics of exact and empirical p-values breaks down. The empirical p-values exhibit systematically reduced volatility compared to true p-values, rendering standard backtesting procedures misleading.

Significance and Claims
The paper claims that the structural mismatch between theoretical PITs and their empirical estimation invalidates the direct applicability of classical goodness-of-fit procedures in standard backtesting contexts. The primary significance of the work lies in identifying that empirical p-values are not independent uniform draws but are instead governed by a two-stage sampling structure.

The author concludes that:

Common-sample implementations effectively convert the problem into a two-sample comparison, requiring revised calibration methods that account for the reference sample's size and variability.
Rolling-window implementations introduce complex dependence structures (autocorrelation) that further distort statistical inference.
Current regulatory and practical backtesting procedures based on PITs require revision to account for these estimation-induced distortions, particularly when reference samples are finite and shared across observations.

The paper does not propose a specific new test statistic but rather calls for a re-evaluation of the null hypothesis distribution in existing frameworks to reflect the underlying two-sample or dependent nature of the data generation process.