📄 other

Nested cross-validation reduces optimism in metabolomics-based prediction of immune checkpoint inhibitor outcomes

This study demonstrates that nested cross-validation significantly reduces optimism and provides more reliable performance estimates than single-split holdout or non-nested cross-validation when developing metabolomics-based predictive models for immune checkpoint inhibitor outcomes.

Original authors: Hirokazu Taguchi, Mirei Shirakashi, Hirotake Tsukamoto, Yuji Miura, Akio Morinobu, Yuki Sugiura

Published 2026-07-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hirokazu Taguchi, Mirei Shirakashi, Hirotake Tsukamoto, Yuji Miura, Akio Morinobu, Yuki Sugiura

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: why do some cancer patients respond incredibly well to a new type of medicine called an immune checkpoint inhibitor (ICI), while others don't? These drugs are like super-charging the body's own police force (the immune system) to hunt down cancer cells. But the police force doesn't always show up to work, and we need a way to predict who will get the help they need. Scientists have started looking at the tiny chemical messengers floating in our blood, called metabolites, to act as clues. This field is called metabolomics. It's like trying to guess the weather by looking at the behavior of ants before a storm. The problem is, these chemical clues are messy, there are thousands of them, and the number of patients we can study is often quite small. When you try to find patterns in a small, messy pile of clues, it's very easy to trick yourself into thinking you've found a perfect pattern when you've actually just found a coincidence. This is the danger of "over-optimism" in science.

This paper is essentially a cautionary tale about how we check our detective work. The researchers wanted to see if the way we test our prediction models makes us think they are better than they really are. They compared three different ways of testing: a simple "holdout" test (where you split the data once into a practice group and a test group), a "non-nested" cross-validation (where you shuffle the data and test it, but accidentally let the clues from the test group influence how you pick the clues), and a "nested" cross-validation (a stricter, more careful method where you keep the test clues completely hidden until the very end).

The study took six different sets of blood samples taken before patients started treatment and ten sets taken after, using high-tech machines to measure chemicals. They ran their prediction models through all three testing methods. The results were a wake-up call. When they used the "non-nested" method, the models looked like rock stars, with an average score (called ROC-AUC) of 0.882 for pre-treatment samples. But when they used the stricter "nested" method, the score dropped to a much more modest 0.705. That's a huge difference! It's like a student getting an A+ on a practice test because they peeked at the answers, but then getting a C on the real exam when the answers were hidden. The "non-nested" method was overestimating the model's ability by nearly 0.18 points on average.

The researchers found that the main culprit for this fake success was the step where the computer picks which chemical clues are the most important. If you let the computer pick the best clues using the same data it uses to grade the test, it gets confused and thinks random noise is a real signal. The "nested" method fixes this by forcing the computer to pick clues on a separate set of data before it ever sees the test data. Interestingly, the "holdout" method (the simple split) didn't overestimate the score as much as the non-nested method, but it was very unstable, giving a wide range of possible answers, like a shaky ruler. The nested method gave a tighter, more reliable range of answers.

The study also looked at whether testing before or after treatment mattered. The answer was: it depends on the specific group of patients. In some cancer types, the blood samples taken before treatment were the best predictors, while in others, the samples taken after treatment started were better. There was no single "magic time" to test everyone.

Finally, the team tested this on a different type of machine (NMR instead of the usual mass spectrometer) with melanoma patients, and the same pattern held up: the strict nested method gave a more realistic, lower score than the looser methods. The authors conclude that for small studies like these, where we are trying to find biomarkers for cancer treatment, we must use the strict "nested" method. If we don't, we might get excited about a "perfect" predictor that is actually just a mirage created by our own testing mistakes. While this doesn't give us a ready-to-use medical test yet, it gives us the right rules for the game so that when we do find a real biomarker, we can trust it.

Technical Summary: Nested Cross-Validation in Metabolomics-Based ICI Prediction

Problem Statement
Immune checkpoint inhibitors (ICIs) improve outcomes for only a subset of cancer patients, necessitating predictive biomarkers. Blood-based metabolomics offers a minimally invasive approach but typically involves small, high-dimensional datasets. In such settings, machine learning models are highly susceptible to overfitting and optimistic performance estimates, particularly when internal validation strategies fail to properly separate model selection from performance evaluation. While non-nested cross-validation (CV) and single-split holdout methods are commonly used, they may inflate apparent discrimination (ROC-AUC) by allowing information leakage during feature selection and hyperparameter tuning. This study addresses the lack of direct comparisons between these validation strategies within the specific context of ICI metabolomics.

Methodology
The authors conducted a multi-dataset methodological study evaluating three internal validation strategies:

Nested Cross-Validation (nCV): Separates model selection (feature ranking, hyperparameter tuning) into an inner loop and performance evaluation into an outer loop.
Non-nested CV: Performs model selection and performance evaluation on the same resampling structure.
Single-split Holdout: Uses a fixed 7:3 training/test split.

Data: The analysis utilized 17 datasets involving patients treated with ICIs across various cancer types (RCC, NSCLC, Melanoma, HNSCC, UC).

LC/MS Datasets: 6 pre-ICI and 10 post-ICI liquid chromatography/mass spectrometry datasets.
NMR Dataset: 1 pre-ICI nuclear magnetic resonance dataset (Melanoma).
Pipeline: A standardized pipeline was applied across all datasets, including preprocessing (imputation, log-transformation, z-scoring), elastic net logistic regression for feature ranking, and L2-regularized logistic regression for classification. Feature selection was restricted to the top $k$ features ( $k \in [1, 10]$ ).

Experimental Design:

nCV: 5-fold outer loop (repeated 5 times) with a 3-fold inner loop (repeated 5 times). All selection steps occurred within the inner loop.
Non-nested CV: Feature ranking and tuning performed on the same 5-fold splits used for evaluation.
Holdout: A single 7:3 split (primary) and 25 repeated splits (sensitivity analysis).
Metrics: Primary metric was ROC-AUC with 95% confidence intervals (CIs) estimated via 2,000 bootstrap resamples. Calibration intercepts and slopes were also assessed.
Ablation Study: To identify the source of optimism, the authors selectively nested only the feature-ranking step (Enet-nest) or only the hyperparameter tuning step (L2-nest) in the pre-ICI datasets.

Key Results

Optimism in Non-nested CV: Non-nested CV consistently overestimated model performance compared to nCV.
- Pre-ICI LC/MS: Mean ROC-AUC was 0.882 (non-nested) vs. 0.705 (nCV), a mean difference of 0.178.
- Post-ICI LC/MS: Mean ROC-AUC was 0.838 (non-nested) vs. 0.646 (nCV), a mean difference of 0.192.
- In Dataset 2 (NSCLC), non-nested CV yielded an ROC-AUC of 0.82, while nCV showed no discriminative ability (0.50).
Uncertainty and Confidence Intervals: Single-split holdout produced significantly wider 95% CIs than nCV, indicating higher instability.
- Pre-ICI: Mean CI width difference (Holdout minus nCV) was 0.313.
- Post-ICI: Mean CI width difference was 0.239.
- Repeated holdout analyses confirmed that nCV still yielded narrower CIs than repeated holdouts.
Source of Optimism: The ablation study revealed that nesting the feature-ranking step (Enet-nest) was the dominant factor in reducing optimism. Enet-nest produced estimates (0.715) nearly identical to full nCV (0.705), whereas nesting only the L2 hyperparameter tuning (L2-nest) resulted in highly optimistic estimates (0.861).
Calibration: Lower ROC-AUC in nCV was not due to conservative probability estimates; calibration slopes were generally closer to the ideal value of 1 for nCV compared to non-nested CV.
Generalizability: The independent NMR dataset (Melanoma) exhibited the same qualitative pattern: non-nested CV (0.847) > nCV (0.696) > Holdout (0.694).
Sampling Time: No single sampling time point (pre- vs. post-ICI) dominated across all cohorts; the most informative time point varied by dataset.

Significance and Claims
The paper claims that internal validation design is a critical determinant of apparent model performance in small, high-dimensional metabolomics studies. The authors assert that:

nCV is the Default Strategy: For metabolomics prediction pipelines involving feature selection and hyperparameter tuning, nested cross-validation should be the standard internal validation method to provide credible estimates of generalization performance.
Optimism is Substantial: The magnitude of optimism in non-nested CV is often large enough to alter the substantive interpretation of a model (e.g., distinguishing a non-informative model from a discriminative one).
Feature Ranking is the Primary Culprit: The leakage of information during the feature-ranking stage is the primary driver of over-optimistic performance estimates in this context.
Translational Context: This study does not propose a clinically deployable biomarker panel but rather provides a validation benchmark. The authors emphasize that robust internal validation (nCV) is a prerequisite for meaningful external validation and clinical translation. They caution that "excellent model performance may reflect the validation design as much as biological signals."

The authors conclude that transparent, prespecified, and fully nested internal validation should be considered standard practice in immuno-oncology metabolomics to ensure reproducibility and prevent the advancement of spurious biomarkers.

Technical Summary: Nested Cross-Validation in Metabolomics-Based ICI Prediction

More like this