Multiple Inputs and Mixwd data for Alzheimer's Disease Classification Based on 3D Vision Transformer

This paper proposes the MIMD-3DVT, a novel 3D Vision Transformer model that integrates multiple brain regions and mixed data sources (imaging, demographics, and cognitive assessments) to achieve a state-of-the-art 97.14% accuracy in classifying Alzheimer's Disease.

The Gist

Imagine your brain is a massive, incredibly complex city. When Alzheimer's disease strikes, it's like a slow-acting fog that starts erasing parts of the city map, destroying neighborhoods, and confusing the traffic lights. Doctors need to spot this fog early to help the city's residents, but looking at the city from just one angle or checking just one street isn't enough.

This paper introduces a new, super-smart "detective" named MIMD-3DVT (a mouthful, so let's call it the 3D Brain Detective) designed to solve this mystery. Here is how it works, broken down into simple concepts:

1. The Problem: The "2D Snapshot" Mistake

Imagine trying to understand a 3D sculpture by looking at a stack of 2D photographs of it. If you look at each photo separately, you might miss how the curves connect or how the shape changes from top to bottom.

• Old Methods: Many previous AI models looked at brain scans like these 2D photos. They analyzed one slice of the brain at a time. This is like trying to guess the plot of a movie by watching just one random frame. You miss the story (the 3D context).
• The "One-Region" Trap: Other models focused only on one specific neighborhood (like the hippocampus, a key area for memory). But Alzheimer's is a city-wide problem; it affects many different areas. Focusing on just one street misses the bigger picture.
• The "Single Clue" Error: Most models relied on just one type of evidence, like only looking at the brain scan. But in real life, doctors also check a patient's age, their test scores, and their history. Ignoring these clues is like a detective ignoring a witness's testimony because they only want to look at the crime scene photos.

2. The Solution: The "All-Seeing" Detective

The authors built a new AI system that acts like a detective who never misses a detail. It has three superpowers:

Superpower A: The 3D Vision (The "Time-Lapse" Camera)

Instead of looking at flat slices, this model looks at the brain as a 3D block of cheese. It processes the slices together, like watching a time-lapse video of the city. This allows it to see how the "fog" (the disease) moves through the 3D space, capturing the shape and structure in a way 2D models can't.

Superpower B: The "Multiple Neighborhoods" Strategy

Instead of checking just one street, the detective sends scouts to six different critical neighborhoods at once:

• The Entorhinal Cortex (the gateway to memory)
• The Fornix (a major information highway)
• The Frontal, Parietal, and Temporal Lobes (areas for thinking, planning, and memory)
• The Hippocampus (the memory center)

By looking at all these areas simultaneously, the model gets a complete map of the damage, not just a partial one.

Superpower C: The "Mixed Bag" of Clues

This is the most unique part. The detective doesn't just look at the brain scan (the image). It also reads the patient's ID card (age, gender) and their report card (cognitive test scores like the MMSE).

• The Analogy: Imagine trying to guess if a car is broken.
  • Old way: You only look at the engine (the MRI scan).
  • New way: You look at the engine, plus you ask the driver how old the car is, plus you check the maintenance log (test scores).
  • Result: You get a much more accurate diagnosis because you have the full story.

3. The Results: A Smarter Diagnosis

The researchers tested this new detective on data from three different major hospitals (datasets).

• The Score: The new model got it right 97.14% of the time.
• The Comparison: Previous "super-smart" models were getting about 96.8% right. While that sounds like a small difference, in the world of medicine, that extra percentage represents many more patients getting the correct diagnosis earlier.
• The Proof: When they added the "mixed clues" (age, test scores) to the brain scans, the accuracy jumped significantly. It proved that combining different types of information is the key to solving the puzzle.

The Bottom Line

This paper is about building a holistic AI doctor. Instead of being a specialist who only looks at one slice of a brain or one type of test, this new system is a generalist that:

1. Sees the brain in 3D (not just flat slices).
2. Checks multiple areas of the brain at once.
3. Listens to the whole patient (scans + age + test scores).

By combining all these perspectives, the "3D Brain Detective" can spot Alzheimer's earlier and more accurately than ever before, giving patients and families a better chance at managing the disease.

Technical Summary

1. Problem Statement

The diagnosis of Alzheimer's Disease (AD) currently faces significant limitations in automated deep learning approaches:

• Loss of 3D Context: Many existing models utilize 2D Transformers that analyze individual MRI slices independently, failing to capture crucial spatial relationships and feature dimensions across consecutive slices.
• Limited Regional Focus: Region of Interest (ROI)-based models often focus on only one or a few brain regions. However, AD affects multiple brain areas simultaneously (e.g., hippocampus, entorhinal cortex, frontal lobes), making single-region analysis insufficient for robust diagnosis.
• Single-Modality Reliance: Most classification models rely solely on imaging data. In clinical practice, AD diagnosis is multifaceted, requiring the integration of demographic factors, cognitive assessments (e.g., MMSE), and imaging. Current models fail to effectively fuse these heterogeneous data types.

2. Methodology

The authors propose a novel framework called Multiple Inputs and Mixed Data 3D Vision Transformer (MIMD-3DVT). The methodology is structured into three main phases:

A. Data Preparation and Preprocessing

• Datasets: The study utilizes a merged dataset from three public repositories: ADNI (Alzheimer's Disease Neuroimaging Initiative), AIBL (Australian Imaging, Biomarker, and Lifestyle Flagship Study), and OASIS (Open Access Series of Imaging Studies).
• Subjects: A total of 420 instances were selected (210 Cognitively Normal [CN] and 210 AD), with 70 subjects per class from each dataset.
• Preprocessing:
  • Skull-Stripping: Removal of non-brain tissues (skin, fat, muscle).
  • Registration: Alignment to the MNI152 T1 template (182 × 218 × 182 resolution, 1mm).
  • Normalization: Numerical data (age, MMSE) scaled to [0, 1]; categorical data (gender) one-hot encoded; images scaled to [0, 1].

B. Instance Selection and ROI Extraction

• Slice Selection: Instead of using full volumes, the method selects specific slices containing ROIs based on the statistical mode of centroid positions (x, y) to ensure the most informative content is extracted.
• Target ROIs: The model processes 3D volumes from six critical regions: Entorhinal Cortex, Fornix, Frontal Lobe, Hippocampus, Parietal Lobe, and Temporal Lobe (both left and right hemispheres).

C. Model Architecture (MIMD-3DVT)

The proposed network is a dual-branch architecture designed to handle mixed data types:

1. Image Branch (3D Vision Transformer):
   • Utilizes ViViT (Video Vision Transformer) to process 3D MRI ROIs.
   • Employs Tubelet Embedding: Instead of treating 3D data as a sequence of 2D patches, it extracts non-overlapping spatio-temporal "tubes" ($t \times h \times w$). This allows the model to capture feature dimensions and spatial information across consecutive slices via self-attention mechanisms.
2. Tabular Branch (MLP):
   • A Multi-Layer Perceptron (MLP) processes categorical and numerical inputs (Age, Gender, MMSE scores).
3. Fusion:
   • The feature vectors from the MLP and the ViViT branches are concatenated to form the final representation for binary classification (CN vs. AD).

D. Training and Validation

• Optimization: Hyperparameters were tuned using Hyperband (an advanced random search with early stopping).
• Validation: A 7-fold cross-validation strategy was employed to ensure robustness and prevent overfitting given the dataset size.
• Metrics: Performance was evaluated using Accuracy, Area Under the Curve (AUC), and Receiver Operating Characteristic (ROC) curves.

3. Key Contributions

1. 3D Spatial Modeling: Introduction of a 3D Vision Transformer that processes consecutive slices together, overcoming the contextual limitations of 2D slice-based analysis.
2. Multi-ROI Fusion: The ability to fuse multiple 3D ROI imaging inputs from various AD-affected brain regions, addressing the limitation of single-region focus.
3. Mixed Data Integration: A novel architecture that effectively integrates heterogeneous data sources (demographics, cognitive scores, and 3D imaging) into a unified transformer model, mimicking the multifaceted clinical diagnostic approach.

4. Experimental Results

The study conducted two primary experiments:

Experiment I: Impact of Mixed Data

• Comparison: "Only-Image" models (ViViT alone) vs. "Mixed Data" models (ViViT + MLP).
• Result: The mixed data approach significantly outperformed the image-only approach.
  • Left Hemisphere: Accuracy increased by 8.18% (from ~90.52% to 98.02%).
  • Right Hemisphere: Accuracy increased by 6.86% (from ~90.52% to 97.38%).
  • Statistical Significance: The improvement was statistically significant with a p-value of 0.001.
• Best Single ROI: The Fornix (Right Hemisphere) achieved the highest single-ROI accuracy of 98.33% when using mixed data.

Experiment II: Performance Comparison with State-of-the-Art

• The proposed MIMD-3DVT was compared against recent literature (e.g., Trans-ResNet, Swin Transformer, standard ViT).
• Results:
  • Multiple ROIs (Merged): Achieved 97.14% accuracy.
  • Single ROIs: Achieved accuracies ranging from 97.86% to 98.33%.
  • Comparison: The proposed method outperformed the best related work (Ref [12], Vision Transformer on ADNI) which achieved 96.80%.
• AUC: The model demonstrated a high average AUC of 0.984 ± 0.011 across 7 folds, indicating excellent discriminative capability.

5. Significance and Future Work

• Significance: This work demonstrates that integrating 3D spatial context with heterogeneous clinical data significantly enhances AD classification accuracy. It provides a more robust, clinically relevant diagnostic tool that mimics the comprehensive nature of human diagnosis.
• Limitations:
  • Dataset Size: The merged dataset (420 subjects) is relatively small for 3D deep learning models, posing a risk of overfitting.
  • Data Modalities: The study is limited to MRI, demographics, and MMSE scores, lacking other clinical modalities like PET scans or blood biomarkers.
• Future Directions:
  • Incorporating additional modalities (PET, segmented gray/white matter, volumetric data).
  • Expanding the dataset size to improve generalization and model robustness.
  • Exploring multi-source domain adaptation to handle shifts between different clinical centers more effectively.

In conclusion, the MIMD-3DVT represents a state-of-the-art advancement in AD classification, proving that the fusion of 3D imaging context with mixed clinical data yields superior diagnostic performance compared to traditional unimodal or 2D approaches.