🤖 machine learning

Universal distribution of the empirical coverage in split conformal prediction

This paper establishes the exact, universal distributions of empirical coverage for split conformal prediction in batch mode, demonstrating that these distributions depend solely on the nominal miscoverage level and calibration sample size to provide a criterion for determining the minimum required calibration data.

Original authors: Paulo C. Marques F

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Paulo C. Marques F

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of modern forecasting, from predicting tomorrow's weather to estimating the price of a house, the most common output is a single number. A model looks at the data and says, "The temperature will be 72 degrees," or "The house is worth $450,000." While these point predictions are useful, they often hide a crucial piece of information: how sure the model is. If a model is guessing wildly, a single number gives a false sense of precision. To fix this, statisticians have developed a framework called conformal prediction. Instead of offering just one answer, this method produces a range of possibilities, or a set of likely outcomes, that comes with a statistical guarantee. It promises that the true answer will fall inside this set a certain percentage of the time, say 95 percent. This approach is powerful because it works with almost any existing prediction tool, requires no specific assumptions about how the data is distributed, and provides these guarantees even when the amount of data is limited.

However, a practical question remains for anyone trying to use these tools: how much data is needed to make the guarantee hold true? In a typical setup, a dataset is split into two parts. One part trains the model, and the other part, known as the calibration sample, is used to tune the size of the prediction set. If the calibration sample is too small, the resulting prediction sets might be too narrow to actually contain the true values, breaking the promised guarantee. If it is too large, the prediction sets become unnecessarily wide, making them less useful. For years, practitioners have had to guess at the right size for this calibration sample, often relying on rough rules of thumb or conservative estimates that might waste valuable data.

A recent study by Paulo C. Marques F. at the Insper Institute of Education and Research in Brazil has solved this problem by finding the exact mathematical behavior of these prediction sets. The researcher focused on a specific version of the method called split conformal prediction, which operates in batches. Imagine a scenario where a model has already been trained and calibrated, and it is now being asked to predict outcomes for a large group of future events all at once. The study asks a simple but profound question: if we look at a batch of future predictions, what is the actual distribution of how many of them turn out to be correct?

The paper establishes that the answer is universal. It does not depend on the specific type of data, the complexity of the model, or the underlying patterns in the world. Instead, the behavior of the prediction sets is determined entirely by two numbers: the desired level of confidence (the nominal miscoverage level) and the size of the calibration sample used to tune the system. The researcher proved that for a finite batch of future observations, the number of correct predictions follows a precise statistical pattern known as a Beta-Binomial distribution. Furthermore, as the batch of future observations grows infinitely large, the proportion of correct predictions settles into a specific pattern known as a Beta distribution. These findings are not merely suggestions or results from computer simulations; they are rigorous mathematical proofs derived from the assumption that the data points are exchangeable, meaning their order does not matter and they come from the same underlying source.

The most significant outcome of this work is a new, precise criterion for choosing the calibration sample size. Because the researcher now knows the exact distribution of the coverage, it is possible to calculate the minimum amount of calibration data required to ensure that the prediction sets are accurate within a specific margin of error, with a high degree of certainty. For instance, if a user wants to be 95 percent sure that their prediction sets are correct within a 5 percent margin of error, the paper provides a direct way to find the exact number of calibration samples needed. The study presents a comprehensive table that lists these minimum sample sizes for various combinations of confidence levels, error margins, and certainty requirements.

This table reveals that the required sample size can vary dramatically depending on how strict the requirements are. For a standard 90 percent confidence level with a generous 10 percent margin of error, a relatively small calibration sample of around 40 to 50 observations might suffice. However, if the user demands a 99 percent confidence level with a very tight 1 percent margin of error, the required sample size jumps to over 40,000 observations. The study clarifies that these numbers are not arbitrary; they are the mathematical minimums derived from the exact distribution of the empirical coverage. This allows practitioners to stop guessing and start calculating, ensuring they use just enough data to meet their reliability goals without wasting resources on excessive calibration.

The paper also addresses a subtle but important distinction in how these predictions behave. While the average performance of the prediction sets is well-known to be close to the target confidence level, the actual performance in any single batch can vary. The study shows that this variation is not random noise but follows a predictable shape. By understanding this shape, users can appreciate that even with a perfect model and a correctly sized calibration sample, there is a natural fluctuation in how many future predictions will be correct. The research confirms that this fluctuation is inherent to the method and is fully characterized by the size of the calibration sample and the chosen confidence level.

In the context of machine learning applications, this work provides a foundational tool for reliability. It transforms the choice of calibration sample size from an art into a science. The researcher demonstrates that the properties of split conformal prediction are robust and universal, holding true regardless of the specific algorithm used to generate the predictions. Whether the task is predicting a continuous value like a stock price or a categorical label like a disease diagnosis, the rules for determining the necessary calibration data remain the same. The study does not claim to improve the accuracy of the underlying models themselves, but rather ensures that the confidence intervals built around those models are mathematically sound and optimally sized.

The findings are particularly relevant for fields where data is expensive or scarce, as they allow users to determine the precise lower bound of data needed to achieve a desired level of trust. Conversely, in scenarios where data is abundant, the study warns against the inefficiency of using unnecessarily large calibration sets, which can lead to overly conservative and uninformative prediction intervals. By providing a clear, universal formula for the distribution of empirical coverage, the paper equips researchers and practitioners with the ability to design their prediction systems with a level of precision that was previously unavailable. The result is a more efficient and reliable framework for quantifying uncertainty in the age of artificial intelligence.

Technical Summary: Universal Distribution of the Empirical Coverage in Split Conformal Prediction

Problem Statement
Split conformal prediction is a widely adopted technique for generating prediction sets with statistical guarantees, balancing predictive performance and computational efficiency. While the marginal validity of these sets is well-established (ensuring that the true label falls within the prediction set with probability at least $1-\alpha$ ), the behavior of the empirical coverage over a finite batch of future observables remains less characterized. Specifically, there is a need to understand the exact distribution of the empirical coverage when applied to a finite batch of future data, as well as the distribution of its limit as the batch size approaches infinity. This understanding is critical for determining the minimum calibration sample size required in practical applications to ensure the empirical coverage remains within a desired tolerance of the nominal level.

Methodology
The paper operates within a supervised learning framework assuming a sequence of random pairs $(X, Y)$ that are exchangeable. The data is partitioned into a training sample, a calibration sample of size $n$ , and a sequence of future observables. The methodology relies on the following core components:

Exchangeability and Conformity Scores: The author utilizes the assumption of data exchangeability. They define a conformity function $\rho$ and associated conformity scores $S_i$ . A key lemma establishes that under the exchangeability assumption, the sequence of conformity scores is itself exchangeable.
Prediction Sets: For a specified nominal miscoverage level $\alpha$ , conformal prediction sets are constructed using the $\lceil(1-\alpha)(n+1)\rceil$ -th order statistic of the calibration conformity scores.
Coverage Indicators: The paper defines binary coverage indicators $Z_i$ for future observables, where $Z_i=1$ if the true response is contained in the prediction set.
Statistical Derivation:
- Finite Batch: By leveraging the exchangeability of the coverage indicators (derived from the exchangeability of conformity scores) and standard combinatorial tools, the author derives the exact distribution of the sum of these indicators for a batch of size $m$ .
- Infinite Batch Limit: The paper applies de Finetti's representation theorem to the sequence of exchangeable coverage indicators to determine the distribution of the almost sure limit of the empirical coverage as the batch size $m \to \infty$ .

Key Contributions and Results

Exact Distribution for Finite Batches (Theorem 1):
The paper derives the exact distribution of the empirical coverage $C_m^{(n,\alpha)}$ for a finite batch of $m$ future observables. The result shows that $m \times C_m^{(n,\alpha)}$ follows a Beta-Binomial distribution with parameters determined by the nominal miscoverage level $\alpha$ and the calibration sample size $n$ . Specifically, the parameters are $\lceil(1-\alpha)(n+1)\rceil$ and $\lfloor\alpha(n+1)\rfloor$ . This distribution is "universal," meaning it depends solely on $n$ and $\alpha$ , not on the underlying data distribution or the specific predictive model used.
Distribution of the Almost Sure Limit (Theorem 2):
As the future batch size $m$ tends to infinity, the empirical coverage $C_m^{(n,\alpha)}$ converges almost surely to a random variable $C_\infty^{(n,\alpha)}$ . The paper proves that this limiting random variable follows a Beta distribution with the same parameters as the finite case: $\text{Beta}(\lceil(1-\alpha)(n+1)\rceil, \lfloor\alpha(n+1)\rfloor)$ .
Universal Criterion for Calibration Sample Size:
Based on the limiting distribution, the author establishes a criterion for selecting the minimum calibration sample size $n$ . Given a nominal level $\alpha$ , a tolerance $\epsilon$ , and a probability threshold $\tau$ , one can calculate the smallest $n$ such that the probability of the empirical coverage deviating from $1-\alpha$ by more than $\epsilon$ is at most $1-\tau$ .

Significance and Practical Application
The paper emphasizes that these results are eminently practical. The derived distributions allow practitioners to move beyond the conservative bounds of marginal validity to a more precise control of empirical coverage.

Universal Tables: The author provides "Universal coverage tolerance tables" (Table 1) that list the minimum required calibration sample sizes for various combinations of $\alpha$ , $\epsilon$ , and $\tau$ .
Comparison with Prior Work: The paper notes that while the necessity of understanding empirical coverage distribution was previously discussed (e.g., in [8]), the values derived in this work are slightly smaller than those in prior literature, offering a more efficient criterion for sample size selection.
Reproducibility: The author provides R code in a repository to reproduce the table values and simulate the results of Theorems 1 and 2, facilitating the application of these theoretical findings in real-world scenarios.

In summary, the paper provides a rigorous, distribution-free characterization of the empirical coverage in split conformal prediction, transforming the selection of calibration sample sizes from a heuristic process into one grounded in exact probabilistic limits.

Technical Summary: Universal Distribution of the Empirical Coverage in Split Conformal Prediction

More like this