📊 statistics

A preregistered test that outcome-permuted data also pass: mathematical coupling in a pan-cancer analysis of proliferation adjustment in prognostic gene expression studies

This paper demonstrates through a preregistered pan-cancer analysis that a hypothesis regarding mathematical coupling in prognostic gene expression studies was confirmed largely by construction, as the test criteria were met even when applied to outcome-permuted null data, highlighting the critical need to validate analytical pipelines against permuted outcomes to avoid false confounding conclusions.

Original authors: SOON WOO NAM

Published 2026-10-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: SOON WOO NAM

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of cancer research, scientists often look for specific genes that act as warning signs for how a patient's disease will progress. By analyzing the activity of these genes in a tumor, researchers hope to predict whether a patient will survive longer or shorter. However, tumors are complex, and their behavior is influenced by many factors, such as the patient's age, the size of the tumor, and how quickly the cancer cells are dividing. A major challenge in this field is distinguishing between a gene that is truly a unique predictor of survival and one that simply reflects the general speed at which the tumor is growing. If a researcher forgets to account for how fast the cancer is dividing, they might mistakenly credit a specific gene with predicting the outcome, when in reality, that gene is just a passenger on the fast-moving train of cell division. This problem is known as mathematical coupling, a situation where two measurements are linked simply because they were calculated using the same underlying data, creating a false sense of connection.

A researcher named Soon Woo Nam set out to test a specific idea about how these gene lists are created and whether standard statistical methods can accidentally confirm a pattern that isn't really there. The study focused on a large collection of data from twenty different groups of cancer patients, covering fourteen types of cancer and nearly five thousand individuals. The goal was to see if adjusting for a "proliferation score"—a measure of how fast the cancer cells are dividing—would change the list of genes that appear to predict survival. The researcher had a preregistered plan, meaning the rules for the analysis were written down before looking at the data, to ensure the process was fair and transparent. The plan predicted that when the speed of cell division was added to the statistical models, the estimated importance of other genes would shift in a very specific, predictable way across all the different cancer types.

When the analysis was run on the actual patient data, the results matched the prediction perfectly. The data showed that the importance of the genes shifted exactly as the plan had forecasted, and this pattern held true for every single one of the twenty cancer groups. At first glance, this might look like a strong confirmation that the statistical method works as intended. However, the researcher then performed a crucial check that is rarely done in such studies. They took the same data and shuffled the survival outcomes—essentially randomizing who lived and who died—so that no gene could possibly be a real predictor of survival. In this scrambled version of reality, where nothing mattered, the same statistical test was run again. Surprisingly, the test still passed. The shuffled data met all the same criteria as the real data, confirming the pattern even though there was no true biological signal to find.

This finding reveals a fundamental flaw in how these tests are sometimes interpreted. The pattern the researcher was looking for is not a sign of a real biological relationship or a specific type of error called confounding. Instead, the pattern is "built-in" to the math itself. Because the new measurement (the speed of cell division) is derived from the same gene data used to calculate the other genes, the statistical models are mathematically forced to show a relationship, regardless of whether that relationship is real or fake. The test was so sensitive to this mathematical structure that it could not tell the difference between a real biological effect and a random shuffle of numbers. In fact, in ninety-two out of one hundred random trials, the test would have passed, even though nothing was actually happening.

The study also looked at how the list of important genes changed when different factors were added to the models. When researchers added the stage of the cancer (how far it had spread) or the speed of cell division to their calculations, the number of genes that appeared to be significant predictors dropped dramatically. In some cases, adding these factors removed more than half of the genes that had previously looked important. This suggests that the list of "prognostic genes" is not a fixed set of biological truths but rather a list that depends heavily on which other factors the researcher chooses to include in their model. If a researcher does not account for how fast the tumor is dividing, they might include many genes in their list that are just reflecting that speed, rather than offering unique information about the patient's future.

The ultimate conclusion is that a preregistered hypothesis can be confirmed simply by the way the data is constructed, not because the hypothesis is true. The researcher demonstrated that running a test on permuted data—where the outcomes are randomized—before finalizing the analysis plan would have exposed this issue immediately. If the test passes when nothing is happening, it cannot be trusted to prove that something is happening when the real data is used. The study serves as a reminder that in complex gene studies, the list of important genes is highly sensitive to the choices made during the analysis. Researchers need to be transparent about which factors they adjust for, and they should understand that changing those factors can drastically alter the results, sometimes removing two-thirds of the genes that initially seemed significant.

Technical Summary: A Preregistered Test That Outcome-Permuted Data Also Pass

Problem Statement
The paper addresses a critical limitation in the utility of preregistration for observational research: while preregistration fixes an analysis plan before data examination, it does not guarantee that the registered test is capable of failing. The author investigates "mathematical coupling," a phenomenon where a quantity is mathematically related to another computed from the same data, which can cause a hypothesis to be confirmed by construction rather than by empirical evidence. Specifically, the study examines prognostic gene expression studies where researchers adjust for tumor proliferation. The author hypothesizes that the observed relationship between the shift in gene hazard ratios (upon adding a proliferation score) and the proliferation score's own hazard ratio is a mathematical artifact inherent to the modeling process, rather than evidence of confounding.

Methodology
The study employs a rigorous, preregistered framework followed by post-hoc null checks using outcome permutation.

Data and Cohorts: The analysis utilized 20 public cohorts (4,986 patients, 14 cancer types) from The Cancer Genome Atlas (TCGA) and the Gene Expression Omnibus (GEO).
Proliferation Score: A standardized proliferation score was constructed as the mean of eight cell-cycle genes (e.g., MKI67, TOP2A) standardized within each cohort.
Registered Analysis Plan (Second Plan):
- For every expressed gene, two Cox proportional hazards models were fitted: Model 1 (gene + age + sex + stage) and Model 2 (Model 1 + proliferation score).
- The "shift" was defined as the factor by which a gene's hazard ratio changed between models.
- Hypotheses: The plan predicted a linear relationship across cohorts between $b$ $b$ (the slope of the log-shift on the gene-score correlation) and $L$ $L$ (the log hazard ratio of the proliferation score).
  - $H_1$ : The coefficient of $L$ is negative (95% CI < 0).
  - $H_2$ : The intercept includes zero.
  - $H_3$ : The sign of $b$ opposes the sign of $L$ in at least 80% of cohorts.
Null Checks (Permutation): To test if the registered criteria could be met by chance or construction, the author permuted survival times and statuses within all 20 cohorts 100 times. This preserves all internal correlations (gene-gene, gene-score) but removes any association with survival. The registered pipeline was re-run on these null datasets.
Covariate Set Comparison: The study compared the number of significant genes under three models: (A) age/sex, (B) age/sex/stage, and (C) age/sex/score, to assess stability.

Key Results

Confirmation of Registered Hypotheses: In the observed data, all three hypotheses ( $H_1, H_2, H_3$ ) were confirmed. The coefficient was $-0.84 $(95% CI$ -0.90$ to $-0.78$), the intercept was near zero, and the sign rule held in 20/20 cohorts.
Failure of the Null Check: Crucially, when the same pipeline was applied to outcome-permuted data (where no gene is prognostic), the criteria were met in 92 out of 100 null replicates (95% CI 85–96).
- The observed coefficient ($-0.842$) fell within the central 95% range of the null distribution ($-1.053$ to $-0.832$).
- The high $R^2$ (0.981) observed in the real data was also indistinguishable from the null distribution (range 0.924–0.990).
Mathematical Coupling: The author concludes that the relationship is "built in." When a covariate (proliferation score) is added to a model, the coefficient of any other variable shifts in proportion to its correlation with the new covariate and the new covariate's own coefficient. This occurs regardless of whether the covariate is a true confounder, a causal mediator, or has no effect at all.
Gene List Instability: The list of significant prognostic genes was highly dependent on the covariate set. Adding stage reduced the number of significant genes from 22,315 to 15,988; adding the proliferation score further reduced this to 7,638. In some cohorts, stage alone removed >99% of significant genes.

Key Contributions

Demonstration of Preregistration Insufficiency: The paper provides a concrete case where a preregistered hypothesis was confirmed not because the biological mechanism was true, but because the statistical test was mathematically coupled to the data structure.
Proposed Remedy (Pre-Registration of Null Behavior): The author proposes a modification to the preregistration workflow: researchers should run the registered pipeline on outcome-permuted data before finalizing the registration. They should then register the values and criteria thresholds derived from these null runs. This ensures the test is "severe" and capable of failing.
Quantification of Covariate Sensitivity: The study quantifies how drastically prognostic gene lists change based on the inclusion of standard covariates (stage) or biological constructs (proliferation scores), highlighting that such lists are not stable properties of the tumor but artifacts of the model specification.

Significance and Claims
The paper modestly claims that the confirmation of the registered hypothesis in the observed data cannot be taken as evidence of confounding by proliferation, as the same result arises in a "null world" where nothing is prognostic.

The primary significance lies in the methodological critique: a preregistered hypothesis can be confirmed largely by construction. The author argues that running a pipeline on permuted outcomes before registering would have exposed this flaw. The paper concludes that lists of prognostic genes are heavily dependent on the covariate set, and researchers must state and vary these sets rather than presenting a single list as definitive. The work does not claim to resolve whether proliferation mediates specific gene effects but rather demonstrates that the statistical method used to detect such mediation is inherently biased by mathematical coupling.

Technical Summary: A Preregistered Test That Outcome-Permuted Data Also Pass

More like this