📄 health informatics

Separation-like irregularity and sample size optimism in high-discrimination logistic prediction models

This study demonstrates that closed-form sample size criteria for logistic prediction models, such as the Riley framework, become increasingly optimistic and underestimate the required sample size as target discrimination rises, primarily due to separation-like behavior that destabilizes calibration, thereby necessitating the use of simulation-based stress tests for high-discrimination scenarios.

Original authors: Liu, Z., Liang, Y., Wang, L. S., Yu, J., Liu, J.

Published 2026-01-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liu, Z., Liang, Y., Wang, L. S., Yu, J., Liu, J.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a chef trying to create a new recipe for a soup that predicts whether a customer will love it or hate it. You have a list of ingredients (predictors) like salt, pepper, and herbs. Your goal is to write down a recipe (a mathematical model) that is so good it can perfectly guess a customer's reaction just by looking at the ingredients.

This paper is about figuring out how many taste tests (data points) you need to write a reliable recipe before you open your restaurant.

The Old Rulebook vs. The New Reality

For a long time, chefs (researchers) used a simple rule of thumb: "If you have 10 ingredients, you need at least 100 taste tests." Later, a more sophisticated rulebook (called the Riley framework, used in a tool called pmsampsize) was created. This rulebook is like a smart calculator that says, "Based on how complex your soup is and how well you think it will taste, here is the exact number of tests you need."

The paper's authors asked a critical question: Is this smart calculator always right?

They discovered that the calculator works great when the soup is "okay" (moderate discrimination). But when the soup is incredibly delicious (high discrimination, meaning the ingredients make the outcome very obvious), the calculator starts to lie. It tells you, "You only need a few tests!" when, in reality, you actually need many more.

The "Perfect Storm" Analogy

Think of it like trying to find a needle in a haystack.

Low Discrimination: The needle is buried deep in a huge haystack. It's hard to find. The calculator says, "You need a lot of time to search." This is correct.
High Discrimination: The needle is glowing neon and sitting right on top of the hay. It's super easy to spot. The calculator looks at this and says, "Wow, that's easy! You only need 5 seconds to find it."

Here is the problem: Just because the needle is glowing doesn't mean you can stop searching after 5 seconds. If you stop too early, you might miss the exact spot where the needle is, or you might think you found it when you actually just saw a shiny piece of foil.

In the world of statistics, when a model is "too good" at predicting the outcome, the math gets shaky. The calculator assumes the math is smooth and easy, but in reality, the numbers start to go wild (a phenomenon called separation). The model starts predicting "100% chance of love" or "0% chance of love" for almost everyone. This looks great on paper, but it's actually a sign that the recipe is unstable and will fail when you try it on new customers.

What the Authors Did

The authors ran a massive computer simulation (a "virtual kitchen") to test this.

They created thousands of fake soups with different levels of "deliciousness" (discrimination).
They asked the calculator how many taste tests were needed.
They then actually ran the tests to see how many were really needed to get a stable, reliable recipe.

The Result:

Moderate Soup: The calculator was right.
Super-Delicious Soup: The calculator was way too optimistic. It suggested sample sizes that were 20% to 40% too small. If researchers followed the calculator's advice for these "perfect" soups, their models would be unstable and their risk predictions would be dangerously inaccurate.

The "Glitch" in the System

Why does the calculator fail? The authors found that when the model is too good, the math inside the computer starts to glitch. It's like a GPS that gets confused when you drive too fast; it thinks it knows the route perfectly, but it's actually spinning in circles.

In their simulations, they saw that at the sample sizes the calculator recommended, the computer models often started producing "extreme" results (predicting 99.999% certainty). Even though the computer said "I'm done, I found the answer," the answer was actually a fluke. The model hadn't truly learned the pattern; it had just memorized the noise.

The Takeaway for Chefs (Researchers)

The paper doesn't say "throw away the calculator." It says: Use the calculator as a starting point, but don't trust it blindly when the soup is too perfect.

If you are building a model that you expect to be very accurate (high discrimination), you should:

Run a "Stress Test": Before you collect all your data, run a small simulation to see if the model starts acting crazy (predicting 100% or 0% too often).
Get More Data: If the stress test shows the model is unstable, you need to collect more data than the calculator suggested.
Use a Safety Net: Instead of using a standard recipe, use a "stabilized" recipe (like Ridge regression) that prevents the model from getting too excited and making extreme predictions.

Summary

The paper warns that when a prediction model is too good at telling the future, the standard math used to plan the study breaks down. It tells researchers to be careful not to be fooled by high accuracy; they need more data and better checks to ensure their model is truly reliable and not just a lucky guess.

Technical Summary: Separation-like Irregularity and Sample Size Optimism in High-Discrimination Logistic Prediction Models

Problem Statement
The development of clinical prediction models, particularly those using logistic regression, relies heavily on adequate sample sizes to ensure stable calibration and avoid overfitting. While the Riley framework (implemented in the pmsampsize R package) provides widely used closed-form criteria for determining minimum sample sizes based on global shrinkage targets, recent evaluations suggest these formulas may become optimistic in specific design regimes. Specifically, there is a concern that when anticipated model discrimination (AUC) is high, the closed-form recommendations may underestimate the sample size required to maintain calibration stability. This paper investigates the operating characteristics of the Riley benchmark ( $n_{Riley}$ ) against an empirically derived sample size requirement ( $n_{req}$ ) defined by out-of-sample calibration-slope stability, with a specific focus on high-discrimination settings where "separation-like" numerical irregularities may occur.

Methodology
The authors conducted a comprehensive Monte Carlo simulation study to compare formula-based recommendations with empirical requirements.

Simulation Design: The study utilized a factorial grid crossing four design factors:
- Predictor Distributions: Normal, standardized skewed continuous (exponential), and binary (Bernoulli).
- Signal Density: Dense (signal distributed across all $p=10$ predictors) versus Sparse (signal concentrated in 3 predictors, 7 noise).
- Target Discrimination: $AUC_{target} \in \{0.70, 0.75, 0.80, 0.85, 0.90\}$ .
- Prevalence: $\phi_{target} \in \{0.05, 0.10, 0.20\}$ .
Data Generation: For each scenario, parameters (intercept $\alpha$ and signal strength $f$ ) were calibrated via nested root-finding on a large synthetic population ( $N=1,000,000$ ) to ensure the generated data matched the target prevalence and AUC exactly.
Benchmark ( $n_{Riley}$ ): Calculated using the pmsampsize package with a global shrinkage target of $S \geq 0.90$ .
Empirical Requirement ( $n_{req}$ ): Defined as the smallest sample size $n$ $n$ such that, under repeated development sampling:
1. The expected logit-scale calibration slope $E(b_n) \geq 0.90$ .
2. The probability of severe miscalibration $Pr(b_n < 0.80) \leq 0.20$ .
- $n_{req}$ was determined via an adaptive bisection search using truth-based calibration slopes evaluated on a large, fixed validation covariate set ( $N_{val}=50,000$ ) to eliminate outcome noise.
Mechanism Diagnostics: At $n = n_{Riley}$ , the study recorded frequencies of non-convergence, separation warnings (extreme fitted log-odds or probabilities), and extreme fitted risks to assess whether numerical pathologies explain discrepancies.
Applied Illustration: A real-world stroke prediction dataset was used to demonstrate the application of these diagnostics and the comparison between unpenalized and ridge-penalized logistic regression.

Key Results

Discrimination-Dependent Divergence: At moderate discrimination ( $AUC_{target} \leq 0.75$ ), $n_{Riley}$ closely approximated $n_{req}$ . However, as discrimination increased, $n_{Riley}$ increasingly underestimated the required sample size. At $AUC_{target} = 0.90$ , the formula-based recommendation was significantly lower than the empirical requirement (e.g., a relative difference of -40.5% to -47.2% depending on prevalence), leading to models with calibration slopes well below the 0.90 target.
Robustness: The primary pattern of underestimation at high discrimination was robust across different predictor distributions (normal, skewed, binary) and signal density configurations (dense vs. sparse).
Mechanism of Optimism: At $n = n_{Riley}$ $n = n_{R i l ey}$ in high-discrimination settings, standard unpenalized logistic regression fits frequently exhibited "separation-like" behavior despite reporting nominal convergence.
- While outright non-convergence was rare, the frequency of separation warnings (extreme fitted risks or log-odds) increased sharply with AUC (reaching ~50% at $AUC=0.90$ for low prevalence).
- This suggests that likelihood-based information measures (e.g., $\Delta\chi^2$ ) used in the Riley shrinkage approximation become inflated or perturbed by near-deterministic predictions, leading to overly optimistic shrinkage estimates and, consequently, insufficient sample size recommendations.
Applied Illustration: In the stroke dataset example, an unpenalized model developed with the formula-recommended sample size showed signs of overfitting (calibration slope 0.802), whereas a ridge-penalized model achieved better stability (slope 1.013), though with a slight shift toward underfitting.

Key Contributions

Empirical Mapping: The study provides a systematic characterization of where closed-form sample size recommendations align with or deviate from simulation-based requirements, identifying target discrimination (AUC) as the primary driver of optimism in high-performance regimes.
Diagnostic Protocol: It operationalizes a set of convergence and separation-related diagnostics (e.g., extreme fitted risks) to be evaluated at the formula-recommended sample size. These diagnostics serve as a practical tool to detect when likelihood-based approximations are strained.
Practical Guidance: The authors propose a workflow where $n_{Riley}$ serves as a principled baseline, but is augmented by targeted simulation stress tests and instability diagnostics when high discrimination is anticipated.

Significance and Claims
The paper claims to offer empirical evidence that the accuracy of formula-based minimum sample size recommendations is a function of anticipated discrimination. The authors assert that while the Riley framework remains a defensible tool for moderate discrimination, its reliance on likelihood-based shrinkage approximations breaks down in high-discrimination settings due to the prevalence of separation-like numerical irregularities.

The significance lies in providing a coherent explanation for why closed-form criteria become optimistic: the fitted likelihood becomes dominated by near-deterministic predictions, weakening the connection between information measures and effective overfitting. Consequently, the authors recommend that researchers planning high-discrimination models should not rely solely on closed-form calculations but should instead employ simulation-based planning or penalized estimation strategies, supported by explicit reporting of calibration stability and numerical diagnostics. The study does not claim to invalidate the Riley framework entirely but rather to define its boundaries of reliability and offer a mechanism-driven approach to mitigate its limitations in specific high-stakes regimes.