Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

Technical Summary: Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

Problem Statement
The field of AI-based neuroimaging has evolved from using hand-engineered anatomical features (e.g., hippocampal volume) combined with classical statistical models to employing complex deep learning architectures like Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). While deep learning models can automatically learn features from vast unlabeled datasets, it remains unclear whether these sophisticated, nonlinear representations offer a significant diagnostic advantage over simpler linear models based on preselected anatomical characteristics. Furthermore, there is a need to understand how effectively foundation models learn anatomical structure during unsupervised pretraining and whether explicit anatomical grounding can improve their performance, particularly in scenarios with limited downstream training data.

Methodology
The authors conducted a comprehensive comparative study across 18 publicly available datasets containing approximately 80,000 participants and over 96,000 3D structural T1-weighted MRI scans. The study evaluated three distinct feature-extraction paradigms across seven clinical tasks (Alzheimer's diagnosis, Parkinson's diagnosis, tumor IDH mutation detection, epilepsy diagnosis, brain age estimation, whole tumor segmentation, and stroke lesion segmentation):

1. Anatomical Linear Models: A linear logistic regression model trained on 178 preselected anatomical features (volumes, surface areas, and cortical thickness) extracted via FreeSurfer.
2. Supervised CNNs: Standard architectures including ResNet-18 for classification and U-Net for segmentation, trained directly on raw MRI volumes.
3. Foundation Models (ViTs):
   • Pretrained ViT: Vision Transformers pretrained on a large general-population cohort (primarily UK Biobank, ~69k scans) using Masked Autoencoding (MAE).
   • Anatomy Segmentation Pretraining (ASP) ViT: A novel pretraining strategy proposed by the authors. This method injects explicit anatomical knowledge during pretraining by adding an auxiliary segmentation task. After initial MAE pretraining, a lightweight decoder is trained to segment 39 distinct brain regions. The encoder and decoder are then trained jointly in alternating epochs to prevent catastrophic forgetting of the MAE features. This approach utilizes a prompt-based architecture to handle multiple regions efficiently without excessive memory overhead.

The models were evaluated using standard metrics (AUC, Dice score, Mean Absolute Error) on internal test sets and multiple external cohorts to assess generalizability.

Key Results

• Parity in Diagnostic Tasks: Across the seven clinical tasks involving datasets ranging from dozens to a few thousand samples, the simple linear model based on anatomical features achieved performance comparable to complex nonlinear models (CNNs and pretrained ViTs). In many cases, the differences in performance were not statistically significant.
• Implicit Anatomical Learning: The study observed that ViTs pretrained via MAE implicitly learn anatomical information, as evidenced by the increasing accuracy of linear probes predicting brain region volumes from the embeddings during pretraining.
• Superiority in Data-Rich Scenarios: When the training data was significantly scaled up (using ~62,890 subjects for biological age estimation), the deep learning models outperformed the linear anatomical model. Specifically, the ASP ViT achieved the best performance, with a Mean Absolute Error (MAE) of 2.54 years on internal tests and 4.19 years on external cohorts. It outperformed the standard Pretrained ViT, particularly on external cohorts and in subjects over 80 years old, a demographic underrepresented in training data.
• Segmentation Performance: For segmentation tasks (tumor and stroke), U-Nets and UNETR generally performed well, though the ASP ViT showed competitive results, particularly when training data was reduced to 10%.

Key Contributions

1. Comparative Benchmarking: The study provides a rigorous, large-scale comparison demonstrating that explicit anatomical features remain highly competitive with state-of-the-art deep learning representations for diagnostic tasks with limited data.
2. Anatomy Segmentation Pretraining (ASP): The authors introduce a novel pretraining framework that explicitly leverages anatomical segmentation as an auxiliary task. This method accelerates the learning of anatomical structure in foundation models.
3. Insight into Feature Learning: The work establishes that while foundation models learn anatomical structure implicitly, explicit guidance via ASP significantly enhances their ability to generalize, especially for tasks requiring high precision (like age estimation) and in underrepresented age groups.

Significance and Claims
The paper claims that for many current neuroimaging diagnostic tasks, the "saturation point" of performance may have been reached within existing public datasets containing hundreds to thousands of scans, where simple anatomical features suffice. The authors suggest that further progress requires substantially larger cohorts. However, they demonstrate that leveraging deep learning tools to extract structured anatomical features provides a computationally efficient and interpretable baseline.

Crucially, the study argues that the ASP framework offers a pathway to mitigate the small size of medical pretraining datasets compared to general computer vision databases. By incorporating clinically relevant tasks (segmentation) during pretraining, the model learns more robust representations that generalize better to downstream tasks, particularly in biological age estimation where the ASP ViT significantly outperformed existing models. The authors conclude that explicit anatomical grounding is a viable and effective strategy to enhance neuroimaging foundation models.