🧬 biology

How reliable is deep learning segmentation of the infant brain? Variability and failure analysis of segmentation pipelines under a common protocol in 51 infants aged 1 to 9 months

This study demonstrates that while three competently configured 3D deep learning pipelines achieve similarly high accuracy in segmenting infant brains with negligible differences driven by retraining or cross-validation, a simple consensus approach effectively eliminates rare segmentation failures, highlighting the critical need to report variability and failure rates rather than relying solely on mean Dice scores.

Original authors: Tasnia Binte Mamun, Samiul Based Shuvo

Published 2026-10-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tasnia Binte Mamun, Samiul Based Shuvo

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The human brain grows faster in the first year of life than at any other time. During this period, the soft tissue inside the skull undergoes a dramatic transformation as it hardens and organizes itself. For doctors and researchers trying to understand how a child develops, or to spot early signs of trouble, seeing inside this changing landscape is essential. They use magnetic resonance imaging, a powerful scanner that creates detailed pictures without using radiation. However, turning these pictures into useful data is difficult. The brain is made of many tiny, folded parts, and in a baby, the contrast between these parts often disappears for a few months, making them look like a uniform gray mass. To study growth, scientists need to count the volume of specific regions, but doing this by hand is slow and prone to human error. Recently, artificial intelligence has stepped in to do the counting automatically, using deep learning to recognize brain structures and draw outlines around them. The hope is that these computer programs can provide a reliable, fast way to measure a baby's brain, but the question remains: can we trust them when the images are so tricky?

A team of researchers set out to test the reliability of these automatic tools on a group of fifty-one infants, ranging from one to nine months old. They gathered a collection of seventy-one brain scans, each containing both T1 and T2 weighted images, which are two different ways of capturing the brain's structure. To ensure a fair test, they did not just look at the final scores; they examined how the tools behaved when the data changed slightly or when the same tool was run multiple times. They compared four different artificial intelligence systems that had been trained specifically on infant data. These systems varied in their design: some used standard three-dimensional blocks to process the image, while others combined different mathematical approaches or looked at the brain in slices. The researchers also created a "consensus" version, where the four systems voted on the final answer for each part of the brain, hoping that combining their opinions would smooth out individual mistakes.

The results showed that the three most advanced three-dimensional systems performed very well, with an average accuracy score of roughly 0.88 out of 1.0. This means they correctly identified the vast majority of the brain tissue. The fourth system, which looked at the brain in slices, performed noticeably worse, with a score around 0.79. However, the differences between the three top performers were so small that they were barely noticeable. In fact, the researchers found that the variation caused by simply changing which babies were in the training group was larger than the difference between the best and second-best systems. This suggests that for these well-configured tools, the specific choice of algorithm matters less than ensuring the data is handled correctly. One system did make a significant mistake on two specific scans, failing to identify several deep brain structures entirely, while the others handled those same scans correctly. This highlighted that even high-performing tools can have rare, catastrophic failures that get hidden when only an average score is reported.

When the researchers combined the predictions of the four systems into a single consensus, the result was even more robust. This combined approach eliminated all the total failures, meaning no brain structures were completely missed, and it slightly improved the overall accuracy. The study also looked at whether the age of the baby or the quality of the image affected the results. They found no evidence that the tools worked better or worse as the infants got older, even during the difficult period when the brain tissue looks most uniform. Similarly, the tools did not seem to struggle more with certain types of image contrast. The researchers concluded that while these artificial intelligence tools are now capable of segmenting infant brains with high accuracy, the way their performance is reported needs to change. Relying on a single average number is not enough; researchers must also report how often the tools fail completely, how much the results vary when the training data changes, and whether the tools agree with each other on a case-by-case basis.

The study serves as a reminder that in the world of medical imaging, a high average score does not guarantee safety or reliability for every single patient. The researchers found that two specific scans were responsible for most of the errors seen in the study, a fact that would have been missed if they had only looked at the group average. By checking for these rare failures and understanding how the tools behave across different groups of data, scientists can build more trustworthy systems. The consensus method, which simply lets multiple tools vote on the answer, proved to be a simple and effective way to remove the worst errors without needing to train a new, complex model. For pediatric radiologists and researchers, the path forward involves not just finding the most accurate tool, but establishing a standard for how to test and report on them, ensuring that the tools used to measure a child's development are as reliable as the science they support.

Technical Summary: Reliability and Variability of Deep Learning Segmentation in Infant Brain MRI

Problem Statement
Deep learning segmentation is increasingly used to derive brain volumes and morphometric data from infant neuroimaging. However, the first postnatal year presents a unique challenge: the "isointense period" (approximately 4–8 months) where grey/white matter contrast is poorly expressed on both T1- and T2-weighted MRI, causing methods designed for older brains to degrade. Current evaluation practices often rely on a single mean Dice score to rank pipelines. The authors argue this approach obscures critical variability, including retraining fluctuations, cross-validation partition effects, and rare but catastrophic per-scan failures that can disproportionately influence cohort means. The study aims to quantify how much observed differences between pipelines reflect genuine performance gaps versus statistical noise and failure modes, and to derive a robust reporting template for validating such tools.

Methodology
The study conducted a retrospective computational evaluation using the public Baby Open Brains (BOBs) dataset, comprising 71 T1- and T2-weighted MRI visits from 51 infants (aged 1.0 to 9.4 months) with 31 expert-corrected reference segmentations.

Pipelines Evaluated: Four distinct deep learning pipelines were compared as end-to-end systems:
1. nnU-Net: A self-configuring 3D CNN.
2. 3D TransUNet: A convolution–transformer hybrid.
3. Swin UNETR: A shifted-window transformer.
4. STSNet: A 2.5D CNN adapted for small targets (using adjacent slice context).
- A cross-fitted consensus was also generated via voxelwise plurality voting of the four out-of-fold predictions.
Experimental Design:
- Subject-Disjoint Cross-Validation: A five-fold split was used where all visits from a single infant were assigned to the same fold to prevent data leakage.
- Subject-Weighted Aggregation: Metrics were averaged within each infant first, then across the 51 infants with equal weighting to prevent infants with repeat visits from dominating the results.
- Metrics: Dice Similarity Coefficient (Dice), Normalized Surface Dice (NSD@1mm), 95th-percentile Hausdorff Distance (HD95), and Average Symmetric Surface Distance (ASSD). Empty predictions were handled via a predefined policy (assigned zero for overlap metrics; excluded for surface metrics).
Statistical Analysis: The study employed paired Wilcoxon signed-rank tests with Holm correction, Cohen's d effect sizes, and linear modeling to assess the contribution of the pipeline term versus other factors (volume, surface-to-volume ratio, age). Retraining variation was measured by repeated runs on identical folds, while partition variation was assessed by comparing fold-specific differences.

Key Results

Performance Accuracy: The consensus pipeline achieved the highest mean Dice (0.8867), followed by 3D TransUNet (0.8805), Swin UNETR (0.8772), and nnU-Net (0.8722). STSNet (2.5D) performed significantly lower (0.7868).
Pipeline Variability: Among the three 3D pipelines, the "pipeline" term explained only 0.134% of the Dice variance ( $p=0.177$ ). All pairwise mean-difference confidence intervals included zero, and standardized effect sizes were small ( $|d| \le 0.22$ ).
Sources of Variance:
- Retraining Variation: Minimal (SD of fold means $\approx 0.0005–0.0016$ ).
- Partition Variation: Significantly larger (SD across folds $\approx 0.0258$ ). The direction of performance differences between nnU-Net and Swin UNETR reversed depending on the fold.
Failure Analysis:
- Rare Failures: Two specific scans drove the majority of failures and statistical anomalies. One scan caused all five empty predictions for nnU-Net; another caused the single empty prediction for 3D TransUNet.
- Mean-Median Disagreement: For the nnU-Net vs. Swin UNETR comparison, the mean difference was negative ($-0.0050$), suggesting Swin UNETR was better, while the median was positive ( $+0.0072$ ) and nnU-Net won in 42 of 51 infants. This reversal was driven entirely by the two influential scans.
Consensus Utility: The cross-fitted consensus eliminated all empty predictions and provided a small mean Dice gain over 3D TransUNet, though the confidence interval for this gain included zero.
Geometric and Age Effects: Dice scores were strongly negatively associated with the surface-to-volume ratio of structures ( $\rho = -0.836$ ). No significant association was found between segmentation performance and chronological age or image contrast, despite the known contrast changes during the isointense period.

Key Contributions

Quantification of Variability: The study demonstrates that for competently configured 3D pipelines, the differences in mean performance are dwarfed by fold-to-fold partition variation and the influence of rare, catastrophic failures.
Failure Mode Identification: It highlights that a single scan can drive cohort-level statistics, leading to contradictory conclusions depending on whether mean, median, or rank-based metrics are reported.
Reporting Template: The authors propose a nine-item reporting template (Table 6) for future benchmarks. This includes requirements for subject-disjoint splits, explicit empty-prediction policies, reporting of retraining/partition variation, and the inspection of failure concentrations.
Consensus Baseline: It establishes that a simple voting consensus can remove total segmentation failures (empty masks) without additional training, serving as a robustness baseline.

Significance and Claims
The paper claims that three widely available, competently configured 3D pipelines can segment the infant brain with high accuracy (mean Dice > 0.87) across the challenging 1–9 month age range, with no detected dependence on age or contrast.

However, the authors maintain a modest and cautious stance regarding the generalizability of these findings:

No Equivalence: The lack of a statistically significant difference between the 3D pipelines is not proof of equivalence, as no equivalence margin was prespecified.
Trust via Evaluation: The authors argue that trust in AI segmentation tools should rest on how they are evaluated (e.g., reporting variability, failure counts, and partition stability) rather than on headline mean scores alone.
Limitations: The study is limited to a single cohort from one site without external validation. The authors explicitly state that the absence of a detected age effect in this sample does not prove such an effect does not exist elsewhere.
Clinical Safety: The study is a research evaluation on a public dataset; the authors make no claims regarding clinical safety or deployment readiness.

In conclusion, the paper suggests that while current 3D deep learning pipelines are promising for infant neuroimaging research, the field must move beyond single-score leaderboards to reporting variability, failure modes, and robustness to ensure reliable downstream morphometric inference.

Technical Summary: Reliability and Variability of Deep Learning Segmentation in Infant Brain MRI

More like this