The practical impact of numerical variability on structural MRI measures of Parkinson's disease

This study demonstrates that numerical variability in neuroimaging pipelines can significantly distort structural MRI findings in Parkinson's disease research, prompting the development of a practical framework to quantify and mitigate these errors to prevent false conclusions in existing literature.

The Gist

Imagine you are trying to measure the height difference between two groups of people: Group A (people with Parkinson's disease) and Group B (healthy people). You want to know if Group A is, on average, slightly shorter than Group B.

To do this, you use a very high-tech, laser-precise ruler (in this case, an MRI scanner and complex computer software called FreeSurfer). You expect the ruler to give you the exact same number every time you measure the same person.

Here is the problem this paper discovered:
Even though the ruler is "precise," the computer doing the math is slightly "jittery."

The "Jittery Calculator" Analogy

Think of your computer like a calculator that is slightly tired or has a shaky hand. When it does a math problem like 2.5 + 2.5, it usually says 5.0. But because of tiny, invisible rounding errors in how computers store numbers (like a digital version of "approximate"), sometimes it might say 4.999999 or 5.000001.

In most everyday math, this doesn't matter. But in brain imaging, scientists are looking for differences that are tiny—sometimes smaller than the width of a human hair. When you run these tiny numbers through a long chain of complex calculations (like measuring brain volume or thickness), those tiny "shakes" in the calculator can get amplified.

By the time the computer finishes its work, the "jitter" might have changed the final result enough to make a healthy brain look like a diseased one, or vice versa.

What the Researchers Did

The team, led by Yohan Chatelain, decided to test this "jitter" on purpose.

1. The Experiment: They took MRI scans of Parkinson's patients and healthy controls. They ran the same scans through the software 26 times.
2. The Trick: Each time they ran it, they added a tiny, random amount of "noise" (simulating the natural differences between different computers, operating systems, or hardware).
3. The Result: They found that the results changed every single time.
   • Sometimes, a specific brain region looked significantly smaller in Parkinson's patients.
   • The next time they ran it (with a tiny bit of digital noise), that same region looked no different between the two groups.
   • In some cases, the "jitter" was responsible for one-third of the total difference they saw between patients and healthy people.

The "Foggy Window" Metaphor

Imagine you are looking at a landscape through a window.

• The Landscape: The real biological differences between a sick brain and a healthy brain.
• The Window: The computer software.
• The Fog: The numerical noise.

The researchers found that the "fog" on the window is so thick in some areas that you can't tell if the landscape has actually changed. You might think you see a mountain (a significant medical finding), but it might just be a trick of the light (numerical noise).

The Big Impact: False Alarms and Missed Cures

Because of this "jitter," the paper found that many previous studies might have made two types of mistakes:

1. False Positives (The Boy Who Cried Wolf): Scientists thought they found a real difference in the brain, but it was actually just a glitch in the computer math.
2. False Negatives (The Missed Clue): Scientists thought there was no difference, but the noise was hiding a real, important biological change that could lead to a cure.

The paper looked at 13 previously published studies on Parkinson's and found that a significant number of their "significant" results were right on the edge of this "jitter." If they had run the numbers on a slightly different computer, those results might have disappeared.

The Solution: A "Noise Detector"

The researchers didn't just point out the problem; they built a tool to fix it.

They created a "Numerical-Population Variability Ratio" (NPVR). Think of this as a Signal-to-Noise Meter.

• The Signal: The real biological difference between patients.
• The Noise: The computer's jitter.

They built a simple, free web tool where scientists can plug in their summary numbers (like "we found a 5% difference"). The tool then calculates: "Is this difference big enough to be real, or is it just as big as the computer's jitter?"

If the "jitter" is as big as the finding, the tool flags it as unreliable.

Why This Matters

This paper is a wake-up call for the entire field of brain imaging. It tells us that computational stability is just as important as biological accuracy.

• Before: Scientists worried about whether their MRI machine was calibrated correctly.
• Now: They must also worry about whether their computer code is "shaky."

By using this new framework, researchers can stop publishing results that are just digital artifacts and start focusing on the real, biological truths about Parkinson's disease. It's about cleaning up the lens so we can finally see the disease clearly.

Technical Summary

1. Problem Statement

Neuroimaging research, particularly in structural MRI, faces significant reproducibility challenges. While much attention has been paid to biological variability, software choices, and preprocessing pipelines, numerical variability (errors arising from floating-point arithmetic, rounding, and truncation) remains understudied.

• The Issue: Computational environments (hardware, OS, library versions) introduce subtle numerical differences that can accumulate during complex processes like image registration and segmentation.
• The Gap: It is unknown whether these minute numerical errors are large enough to alter statistical conclusions in clinical studies, specifically regarding Parkinson's Disease (PD), where subtle morphometric differences are critical.
• The Consequence: If numerical noise is comparable to biological signal, it can lead to false positives (Type I errors) or false negatives (Type II errors), undermining the reliability of biomarkers.

2. Methodology

The authors employed a two-pronged approach: empirical perturbation to quantify the problem and analytical modeling to create a scalable solution.

A. Empirical Quantification (Monte Carlo Arithmetic)

• Dataset: Used T1-weighted MRI data from the Parkinson's Progression Markers Initiative (PPMI), comprising 112 PD patients (without Mild Cognitive Impairment) and 89 Healthy Controls (HC).
• Pipeline: Processed data using FreeSurfer 7.3.1.
• Perturbation Technique: Instrumented FreeSurfer with Fuzzy-libm, a library using Monte Carlo Arithmetic (MCA). MCA injects controlled, zero-mean random noise into floating-point operations (simulating IEEE-754 machine precision variations) without changing the mathematical expectation.
• Execution: Each scan was processed 34 times under different numerical perturbations. After quality control, 26 valid runs per subject were retained to estimate numerical variability ($\sigma_{num}$).
• Analysis: Conducted cross-sectional (baseline) and longitudinal (rate of change) analyses for:
  • Group differences (PD vs. HC).
  • Partial correlations between regional volumes/thickness and UPDRS-III motor scores.

B. Analytical Framework (NPVR)

To avoid the computational cost of re-running MCA for every study, the authors derived a closed-form analytical model:

• Metric: Introduced the Numerical-Population Variability Ratio (NPVR or $\nu_{npv}$):
  $$\nu_{npv} = \frac{\sigma_{num}}{\sigma_{pop}}$$
  Where $\sigma_{num}$ is numerical variability and $\sigma_{pop}$ is inter-subject biological variability.
• Uncertainty Propagation: Using the delta method, they derived formulas to propagate $\nu_{npv}$ through common statistical estimators (Cohen's $d$, $t$-tests, ANCOVA, Partial Correlation) to estimate the uncertainty in effect sizes and $p$-values.
• Significance Flip Probability: Modeled reported $p$-values as a Beta distribution centered on the reported value with variance derived from the analytical model. This allowed the calculation of the probability that a result would flip from significant to non-significant (or vice versa) due to numerical noise.

C. Retrospective Application

The framework was applied to 13 previously published PD MRI studies using only their reported summary statistics to estimate the prevalence of numerically induced false positives/negatives in the literature.

3. Key Contributions

1. Quantification of Impact: Demonstrated that numerical variability in FreeSurfer can reach up to 80% of the population variability in certain regions, altering statistical inferences.
2. The NPVR Metric: Established a standardized metric to compare computational instability against biological signal, enabling direct comparison across pipelines and cohorts.
3. Analytical Tooling: Developed a closed-form framework (and an open-source web tool) that allows researchers to estimate numerical uncertainty and the risk of "significance flips" using only summary statistics, without needing raw data or re-running pipelines.
4. Longitudinal Instability: Identified that longitudinal analyses (subtracting time points) are significantly more susceptible to numerical noise due to catastrophic cancellation, leading to much higher instability than cross-sectional analyses.

4. Key Results

• Statistical Instability: In the empirical study, 27% of subcortical volume comparisons and 21% of cortical thickness comparisons changed their significance status ($p < 0.05$) across the 26 MCA repetitions.
• Magnitude of Noise:
  • Cross-sectional: Average $\nu_{npv} \approx 0.18$ (PD) and $0.17$ (HC).
  • Longitudinal: Average $\nu_{npv} \approx 0.56$ (PD) and $0.55$ (HC).
  • Implication: To reduce numerical uncertainty to a negligible level in longitudinal studies would require a sample size exceeding 12,000 participants, compared to ~1,340 for cross-sectional studies.
• Literature Impact: Applying the model to 13 published studies revealed that findings near the significance threshold are highly fragile. A substantial fraction of reported significant results have a non-negligible probability of being numerically induced false positives.
• Spatial Variability: Numerical instability varied by region. Cortical thickness showed the highest relative precision, while surface area and volume were less stable. Subcortical structures generally showed higher spatial robustness (Dice coefficients) than cortical parcellations.

5. Significance and Implications

• Reproducibility Crisis: This work provides a concrete computational mechanism explaining persistent reproducibility issues in neuroimaging. It suggests that "noise" in results is not just biological or methodological but also numerical.
• Clinical Reliability: For diseases like Parkinson's, where biomarkers are subtle, numerical noise can obscure true clinical associations or create spurious ones.
• Future Standards: The authors argue for the systematic evaluation of numerical stability as a standard part of neuroimaging quality control.
• Scalability: The NPVR framework allows the community to retrospectively audit existing literature and design future studies with adequate power to overcome numerical noise, moving beyond the "black box" of computational pipelines.
• Deep Learning Context: The paper notes that while deep learning inference may be stable, the training process introduces new sources of numerical variability, making this framework relevant for the next generation of neuroimaging tools.

In conclusion, the paper establishes that numerical variability is a substantive factor in neuroimaging, capable of flipping statistical conclusions, and provides a practical, scalable framework to measure and mitigate this risk.