Impact of simulated MRI artifacts on deep learning-based brain age prediction

This study demonstrates that simulated MRI artifacts significantly and differentially degrade the accuracy of deep learning-based brain age prediction across three algorithms (Pyment, MIDI, and MCCQR), highlighting the critical need for artifact-aware evaluation and mitigation strategies in clinical research.

The Gist

Imagine your brain is a unique, intricate house. Scientists have developed a special "AI Architect" that looks at photos of this house (MRI scans) and tries to guess how old the house really is, based on how worn down the paint is or how many cracks are in the walls. This guess is called "Brain Age."

If the AI says your brain is 50 years old, but you are actually 40, it might mean your brain is aging faster than it should—a sign of potential health issues.

However, there's a catch: The quality of the photo matters.

This paper is like a stress test for three different AI Architects. The researchers wanted to see: What happens if the photo of the brain house is blurry, shaky, has static noise, or looks like a ghostly double-image?

The Experiment: The "Photo Booth" Test

The researchers took perfect, high-quality photos of 293 healthy people's brains. Then, they used a computer program to intentionally "ruin" these photos in four specific ways, creating 10 levels of damage for each:

1. Motion: Like someone sneezing or moving their head while the camera took the picture.
2. Ghosting: Like a camera glitch where the image is repeated or smeared, creating a "ghost" of the brain next to the real one.
3. Blurring: Like taking a photo with a dirty lens or out of focus.
4. Noise: Like old TV static or graininess in the photo.

They then fed these "ruined" photos into three popular AI brain-age predictors to see how they reacted.

The Three AI Architects

The study compared three different "Architects" (algorithms), each with a different background:

• Pyment: Trained mostly on perfect, research-grade photos (like photos taken in a super-controlled studio).
• MIDI: Trained on real-world hospital photos (which often have more imperfections).
• MCCQR: A sophisticated architect that tries to calculate its own "confidence" in the answer.

The Results: Who Crumbled and Who Stood Tall?

Here is what happened when the photos got worse:

1. The "Studio" Architect (Pyment) Crumbled Fast
Because Pyment was trained only on perfect photos, it was very fragile.

• The Metaphor: Imagine a chef who has only ever cooked in a pristine, sterile kitchen. If you hand them a slightly dirty pan or a slightly burnt ingredient, they panic and ruin the meal.
• The Result: Even a tiny bit of motion or ghosting made Pyment's guesses wildly inaccurate. It started guessing ages that were years off, and its predictions became inconsistent (sometimes guessing 40, then 50 for the same person).

2. The "Hospital" Architect (MIDI) Was Tougher
MIDI was trained on messy, real-world data.

• The Metaphor: This chef has cooked in busy, noisy kitchens with dirty pans. A little bit of smoke or a smudge on the lens doesn't bother them; they know how to adjust.
• The Result: MIDI handled motion and ghosting much better. It kept giving consistent answers even when the photos were quite bad. However, it did struggle a bit with heavy "static noise."

3. The "Confident" Architect (MCCQR) Was a Mixed Bag

• The Metaphor: This architect is very smart and knows when it's unsure. It kept its ranking of people correct (knowing who is older than whom) even with bad photos, but the actual number it gave (the age) sometimes jumped around wildly when the damage was extreme.

The Big Takeaways

• Motion and Ghosts are the Worst Enemies: Just like a shaky camera ruins a photo, head movement and "ghost" artifacts confused the AI the most, making it guess ages that were completely wrong.
• Blur and Noise are Less Dangerous: Surprisingly, a slightly blurry photo or a grainy one didn't confuse the AI as much as motion did. The AI seems to rely more on the overall shape of the brain than on tiny, sharp details.
• Training Matters: If you train an AI only on perfect data, it will fail in the real world. If you train it on messy, real-world data (like hospitals), it becomes much more robust.
• The "Age" Bias: The study found that for some algorithms, the errors got worse for older people. It's like a ruler that stretches more as the object gets bigger.

Why Should You Care?

If doctors start using "Brain Age" to diagnose diseases like Alzheimer's, they need to know if the result is real or just a glitch caused by a shaky scan.

• The Warning: If a patient moves their head slightly during a scan, a fragile AI might tell them their brain is "10 years older" than it really is. This could cause unnecessary panic or misdiagnosis.
• The Solution: We need AI that is trained on messy, real-world data (like MIDI) and we need to be careful about how we take the pictures. We can't just trust the number the computer gives us; we have to check if the photo was good enough to trust that number.

In short: Your brain age is a useful tool, but only if the picture of your brain is clear. If the photo is shaky, the AI might get confused, and different AIs handle the confusion very differently.

Technical Summary

1. Problem Statement

Brain age prediction is a promising biomarker for detecting atypical aging and neurodegenerative diseases. However, the clinical utility of these deep learning (DL) models is threatened by the variability in MRI image quality found in real-world clinical settings. While most algorithms are trained and validated on high-quality, research-grade data, clinical scans often contain artifacts (motion, ghosting, blurring, noise) that can compromise prediction accuracy.

The core problem addressed is the lack of systematic understanding regarding:

• How specific types and severities of MRI artifacts affect the accuracy and stability of different DL-based brain age algorithms.
• Whether algorithms trained on research-grade data generalize to degraded clinical-grade images.
• How different algorithmic architectures and preprocessing pipelines influence robustness to these artifacts.

2. Methodology

Dataset and Participants:

• Source: The ABRIM dataset (Advanced Brain Imaging on ageing and Memory), containing T1-weighted MPRAGE scans from 293 healthy adults (ages 18–85).
• Acquisition: 3T Siemens Magnetom Prisma, 0.8mm isotropic resolution.
• Baseline: All original images were confirmed artifact-free (Severity Level 0).

Artifact Simulation:

• Tool: Open-source Python toolbox TorchIO.
• Artifacts Simulated: Motion, Ghosting, Blurring, and Noise.
• Severity Levels: 10 distinct levels (1–10) generated using a power-function mapping of simulation parameters.
• Calibration: Severity levels were calibrated against the PondrAI QC visual rating scale (1=Perfect to 6=Terrible). Levels 1–7 correspond to "perfect/good," while 8–10 correspond to "bad/terrible."
• Total Data: 293 original images + 2,930 degraded images (4 artifact types × 10 levels × 293 subjects).

Algorithms Evaluated:
Three widely used, pre-trained deep learning algorithms were tested:

1. Pyment: Trained on 34k research subjects (UK Biobank heavy). Uses a Simple Fully Convolutional Network (SFCN). Preprocessing: FreeSurfer skull-stripping, MNI registration.
2. MIDI: Trained on 2.3k clinical scans (radiologically normal). Uses DenseNet121. Preprocessing: HD-BET skull-stripping, ANTs registration.
3. MCCQR: Trained on 10k healthy subjects (German National Cohort). Uses a multi-channel 3D CNN with Monte Carlo dropout and Composite Quantile Regression. Preprocessing: CAT12 toolbox.

Statistical Analysis:

• Models: Linear mixed-effects models (LMM) with participant as a random intercept.
• Metrics:
  • Deviations: Difference in predicted brain age vs. artifact-free baseline.
  • Performance: Pearson's $R$, $R^2$, Mean Absolute Error (MAE), RMSE (expressed as % change from baseline).
  • Stability: Intraclass Correlation Coefficient (ICC) and Within-subject Coefficient of Variation (wsCV).
  • Clinical Relevance: Percentage of predictions within a Maximal Allowable Difference (MAD) of ±3.55 years.
• Controls: Analyses were repeated with age-bias correction and stratified by age groups (Young, Middle, Older).

3. Key Results

Algorithm-Specific Sensitivity:

• Pyment: Showed the highest sensitivity to artifacts. Motion and ghosting caused significant deviations from severity level 1. At maximum severity, MAE increased by 110% (motion) and 75% (ghosting). Preprocessing failures were frequent (up to 321 images lost for noise), particularly for blurring and noise.
• MCCQR: Demonstrated the best baseline accuracy (MAE 4.6 years) and high stability at low severities. However, it suffered massive MAE inflation at maximum severity for motion (+112%) and ghosting (+135%), though correlations remained relatively stable.
• MIDI: Showed the most robust stability. While its baseline MAE was higher (9.1 years), it maintained consistent predictions across artifacts. MAE increases were lower for motion (+16%) and ghosting (+34%). It had 0% preprocessing failures.

Artifact-Specific Effects:

• Motion & Ghosting: Caused the most severe degradation in both accuracy and stability. These structured spatial distortions disrupted anatomical integrity, leading to large deviations in brain age estimates.
• Blurring: Had minimal impact at low-to-moderate levels. Significant degradation only occurred at maximum severity (Level 10), where MCCQR MAE increased by 137%.
• Noise: Generally had the least impact. Pyment and MCCQR were largely unaffected (MAE increase ≤20%), while MIDI showed a moderate decline (MAE +35%).

Stability and Age Effects:

• Stability: Pyment's ICC dropped from "excellent" to "poor" under motion. MIDI and MCCQR maintained "good" to "excellent" ICC for most conditions, though both degraded at extreme severities.
• Age Bias: Age-bias correction significantly reduced the apparent sensitivity of Pyment, suggesting its artifact sensitivity was partly driven by age-dependent prediction bias.
• Age Stratification: Older adults generally showed larger performance degradation for motion and ghosting across algorithms, likely due to reduced tissue contrast and partial volume effects.

4. Key Contributions

1. Systematic Quantification: Provided the first systematic assessment of how four major artifact types across ten severity levels impact three distinct, widely used brain age algorithms.
2. Algorithm Comparison: Demonstrated that training data origin (research vs. clinical) and architecture (SFCN vs. DenseNet vs. Quantile Regression) are critical determinants of robustness.
   • MIDI (clinical-trained) showed superior stability against motion/ghosting.
   • Pyment (research-trained) was highly fragile, with high preprocessing failure rates.
3. Preprocessing Vulnerability: Highlighted that preprocessing pipelines (e.g., FreeSurfer skull-stripping) are often the first point of failure under artifact conditions, leading to data loss that biases performance metrics.
4. Clinical Thresholds: Identified that even "visually negligible" artifacts (Severity 1–4) can push brain age predictions outside clinically interpretable bounds (MAD > 3.55 years), particularly for research-optimized models.

5. Significance and Implications

• Clinical Translation: The study warns against directly applying research-grade brain age models to clinical data without validation. Algorithms trained on pristine data (like Pyment) may produce false positives (accelerated aging) in the presence of common clinical artifacts.
• Model Selection: For clinical environments where image quality varies, models trained on diverse clinical datasets (like MIDI) or those with robust architectures (like DenseNet) may be preferable, despite potentially higher baseline error.
• Quality Control: Emphasizes the need for "artifact-aware" evaluation. Standard QC metrics may not be sufficient; models must be tested against specific degradation types.
• Future Directions: Calls for the development of training datasets that include annotated artifacts and the creation of "quality-agnostic" algorithms. It also suggests that future simulations should move beyond 2D/slice-wise approaches to 3D anatomically realistic simulations to better capture real-world motion complexity.

Conclusion:
Brain age prediction is highly sensitive to MRI artifacts, but this sensitivity is not uniform across algorithms. The choice of algorithm significantly dictates the reliability of the biomarker in clinical settings. To ensure clinical utility, future models must be trained on diverse, clinically representative data and accompanied by robust preprocessing pipelines capable of handling real-world image degradation.