📊 statistics

Optimal Hold-Out Size in Cross-Validation

This paper proposes a principled, utility-based framework for selecting the optimal hold-out size in cross-validation by explicitly balancing training sample size against evaluation uncertainty, thereby replacing arbitrary conventions with context-specific choices that improve the reliability of model selection and downstream scientific inference.

Original authors: Kenichiro McAlinn, Kōsaku Takanashi

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kenichiro McAlinn, Kōsaku Takanashi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Goldilocks" Dilemma of Testing

Imagine you are a chef trying to perfect a new soup recipe. You have a pot of ingredients (your data). To know if the soup tastes good, you have to taste it. But here is the catch: You can't taste the soup before you finish cooking it, but you also can't cook it perfectly if you keep stopping to taste it.

In data science, this is called Cross-Validation. You split your data into two piles:

The Training Pile: Used to teach the model (the chef) how to make the soup.
The Testing Pile (Hold-out): Used to taste the soup and see if it's actually good.

The paper argues that for years, scientists have been guessing how big these piles should be. They usually just pick a standard rule, like "80% for training, 20% for testing" (or splitting it into 5 or 10 chunks). The authors say this is like a chef blindly guessing how much soup to save for tasting without knowing if they have enough ingredients to cook a good batch first.

The Trade-Off: Cooking vs. Tasting

The paper identifies a tug-of-war between two competing needs:

Need More Training (Small Test Pile): If you give the chef almost all the ingredients to practice with (a tiny test pile), they will learn the recipe very well. The soup will likely taste great. But, because you only have a tiny spoonful to taste, you aren't sure if that one spoonful represents the whole pot. It might be a lucky spoonful or a bad one. Your "certainty" is low.
Need More Testing (Large Test Pile): If you save a huge bowl of soup to taste, you can be very sure about how the soup tastes. Your "certainty" is high. But, the chef had to cook with fewer ingredients, so the soup itself might be under-seasoned or poorly made. The model is less accurate.

The Paper's Goal: Find the "Goldilocks" spot—the perfect size for the test pile that balances a well-cooked soup with a reliable taste test.

The Secret Ingredient: "Irreducible Noise" ( $\sigma^2$ )

The authors discovered that the answer depends on something called Irreducible Noise.

Think of this as the "chaos" or "randomness" in your ingredients.

Low Noise: Imagine baking a cake in a perfect lab where temperature and humidity are controlled. The ingredients behave exactly as expected. The "noise" is low.
High Noise: Imagine trying to predict the weather or human behavior. There is so much randomness that even a perfect model can't be 100% sure. The "noise" is high.

The paper claims that how much you should trust your test results depends on how noisy your data is.

If your data is clean and predictable (Low Noise), you can afford to give the model more data to learn (make the test pile smaller) because the results are stable.
If your data is messy and chaotic (High Noise), you need a bigger test pile to be sure the model isn't just guessing, even if that means the model had less practice.

The Solution: A "Tuning Knob" Instead of a Rulebook

Instead of saying "Always use 5 folds," the authors propose a new method where the researcher acts like a radio tuner.

Estimate the Noise: You have to decide (or guess) how noisy your data is. This is your "tuning knob."
Run the Math: The paper provides a formula that takes your noise level and tells you the optimal size for your test pile.
The Result: You get a map. It might say, "If your data is very noisy, use a 2-fold split. If it's clean, use a 20-fold split."

Real-World Examples from the Paper

The authors tested this on real data to prove that the "one-size-fits-all" rule is wrong.

The Abalone (Sea Snail) Example: They tried to predict the age of sea snails.
- For a simple model (Linear Regression), the best test size changed a lot depending on how noisy they assumed the data was.
- For a complex model (Random Forest), the best test size was different again.
- The Lesson: Using the same test size for both models was misleading. The complex model needed more practice (smaller test pile), while the simple model needed more certainty (larger test pile).
The Cancer Mutation Example: They looked at genetic data to find patterns in cancer.
- When they changed the test size (the "K" value), the list of "important" cancer genes changed.
- The Lesson: A tiny change in how you split the data can change the scientific conclusion about which genes are causing the disease.

Why This Matters for Everyone

The paper concludes that there is no single "best" number for splitting data.

For Scientists: Stop blindly using "5-fold" or "10-fold" cross-validation. You need to ask: "How noisy is my data?" and "How complex is my model?"
The Takeaway: The choice of how to split your data is a scientific decision, not just a computer setting. By making the assumption about "noise" explicit, researchers can avoid drawing false conclusions.

In short: The paper gives scientists a new tool to stop guessing and start calculating the perfect balance between teaching a model and testing it, ensuring that their scientific discoveries are actually real and not just a fluke of how they sliced the data.

Technical Summary: Optimal Hold-Out Size in Cross-Validation

Problem Statement

Cross-validation (CV) is a standard procedure for model selection and parameter tuning, yet the choice of the hold-out size (or equivalently, the number of folds $K$ ) is typically governed by convention (e.g., $K=5, 10$ , or an 80/20 split) rather than a principled criterion. This paper identifies a fundamental tradeoff inherent in the CV procedure: increasing the training sample size ( $n$ ) improves model accuracy (reducing predictive bias), while decreasing the test sample size ( $m$ ) increases the variance (uncertainty) of the performance estimate.

In many modern contexts—particularly with flexible models, high-dimensional data, or flat performance curves—small changes in $K$ can lead to different selected models and divergent scientific conclusions, even when estimated losses are nearly identical. The core problem is that evaluation uncertainty in $K$ -fold CV is not universally estimable from observed data without additional assumptions (Bengio and Grandvalet, 2004), leaving researchers without a general, finite-sample, dataset-specific rule to balance accuracy against evaluation certainty.

Methodology

The authors formalize the selection of the hold-out size as a decision problem targeting pure predictive performance (performance net of irreducible noise) rather than raw empirical loss. The methodology proceeds through four main steps:

1. Theoretical Characterization of Uncertainty

The paper derives finite-sample expressions for the variance of the unbiased estimator of pure loss.

Symmetric Errors: Under the assumption that errors $\epsilon_i$ are symmetrically distributed (e.g., Gaussian), the authors derive an exact expression for the variance of the loss.
Asymmetric Errors: For general cases (e.g., Gamma distributions, binary outcomes), they derive a conservative upper bound.
Key Insight: The variance of the loss is shown to be a function of the model bias, the irreducible error ( $\sigma^2$ ), and the hold-out size ( $m$ ). Specifically, the variance scales with $1/m$ and the squared model bias. The authors note that while $K$ -fold CV averages correlated losses, the variance of a single fold serves as a valid, conservative upper bound for the full $K$ -fold estimate.

2. Mean-Variance Utility Framework

To determine the optimal tradeoff, the authors adopt a von Neumann-Morgenstern expected utility framework. They define a utility function that penalizes both the expected loss and the variance of that loss:
$\text{Utility}(n, m) = - \left( E[\hat{L}_k] + \text{Var}[\hat{L}_k] \right)$
This formulation treats the irreducible noise parameter $\sigma^2$ as a tunable "knob" reflecting the analyst's tolerance for evaluation uncertainty.

3. Estimation Strategy

The proposed strategy to find the optimal $m$ (and thus $K$ ) involves:

Loss Curve Estimation: Estimating the empirical prediction loss at three anchor points: Leave-One-Out CV ( $m \approx 1$ ), Leave-Most-Out CV ( $m \approx N/2$ ), and an intermediate point (e.g., 5-fold). These points are used to interpolate a loss curve $E\hat{L}(m)$ as a function of test set size.
Variance Curve Calculation: Using the derived theoretical bounds, the variance curve $V\hat{L}(m)$ is calculated as a function of the estimated loss and the assumed $\sigma^2$ .
Optimization: The optimal hold-out size is the value of $m$ that minimizes the sum of the estimated loss and its variance bound (maximizing utility).

4. The Pareto Frontier

By varying the assumed irreducible noise $\sigma^2$ , the authors construct a Pareto frontier. This map illustrates the optimal $K$ for different noise regimes, allowing researchers to visualize how the optimal split changes based on their assumptions about data quality.

Key Results

Empirical analyses across multiple domains (biology, materials engineering, social science) and models (linear regression, random forests) yield the following findings:

Context-Dependence: The optimal $K$ is not universal; it varies significantly based on the specific dataset, the model class, and the assumed level of irreducible noise.
Model Complexity: Complex models (e.g., Random Forests) generally require larger training sets (smaller $K$ ) to reduce bias, whereas simpler models (e.g., Linear Regression) can tolerate larger test sets (larger $K$ ) to reduce evaluation variance.
Failure of Conventions: Conventional choices (e.g., $K=5$ ) are often suboptimal. In several experiments, $K=5$ was never the optimal choice for complex models, and for simple models, it was only optimal under specific noise assumptions.
Reverse Engineering $\sigma^2$ : The framework allows researchers to "reverse engineer" the implicit noise level assumed by a conventional split. The authors found that standard splits often imply noise levels inconsistent with the observed model performance (e.g., implying $\sigma^2$ is too high for the data to support the observed low error), suggesting that conventional $K$ choices may be inappropriate for specific problems.
Scientific Impact: In a high-dimensional genomics application (TCGA mutational signature attribution), the choice of $K$ altered the set of "reliably present" mutational signatures. High-uncertainty regimes (favoring small $K$ ) excluded signatures that were detected under low-uncertainty regimes (favoring larger $K$ ), demonstrating that $K$ acts as a lever on sparsity and downstream scientific inference.

Significance and Claims

The paper claims to replace the "one-size-fits-all" convention of cross-validation with a context-specific, assumption-explicit framework. Its primary contributions are:

Explicit Tradeoff: It makes the implicit tradeoff between training accuracy and evaluation certainty explicit, providing a transparent utility-based rule for selecting $K$ .
Finite-Sample Theory: It provides exact finite-sample variance expressions (under symmetry) and general upper bounds, addressing a gap where previous theory offered only asymptotic descriptions or specific-case guidance.
Diagnostic Tool: By treating $\sigma^2$ as a parameter, the framework enables sensitivity analysis. Researchers can identify robust ranges of $K$ where conclusions hold across different noise assumptions or detect when a conventional split implies unrealistic noise levels.
Scientific Rigor: The authors argue that since CV is often part of the inferential pipeline (determining which variables or mechanisms are "supported"), the choice of $K$ must be a scientific decision aligned with domain knowledge about data quality, rather than a default computational setting.

The paper concludes that while exact optimal $K$ values are difficult to pin down universally, the framework demonstrates that optimal splits rarely align with extreme conventions (like LOOCV or 2-fold) and are rarely fixed at standard values like 5 or 10 across all contexts. Instead, researchers should identify reasonable ranges of hold-out sizes tailored to their specific data and model characteristics.