⚡ electrical engineering

Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data

This paper proposes a Multi-Plane Vision Transformer (MP-ViT) that utilizes separate encoders and cross-attention mechanisms to effectively integrate axial and sagittal MRI data for brain hemorrhage classification, achieving superior performance over existing ViT and CNN models on a large clinical dataset without requiring resampling.

Original authors: Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Badhan Kumar Das, Gengyan Zhao, Boris Mailhe, Thomas J. Re, Dorin Comaniciu, Eli Gibson, Andreas Maier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify a specific type of stain on a complex, 3D piece of fabric. To see the stain clearly, you might need to look at it from the top (axial view) and from the side (sagittal view). Sometimes, the stain is obvious from the top but hidden from the side, or vice versa.

In the medical world, doctors use MRI scans to look for brain hemorrhages (bleeds). However, these scans are tricky. They come in different "orientations" (like looking at the brain from the top or the side) and sometimes, a doctor might only have the top view or only the side view for a specific patient.

The Old Way: Forcing a Square Peg into a Round Hole
Traditionally, computer programs (AI) that analyze these images are picky. They usually demand that every image be flattened and resized to look exactly the same, as if you were squishing a 3D object into a flat 2D photo.

The Problem: If you take a side-view MRI and force it to look like a top-view MRI, you have to stretch or squish the pixels. The paper calls this "resampling." It's like trying to fit a tall, thin vase into a short, wide box by smashing it down. You lose important details, and the AI might miss the hemorrhage because the image got distorted.

The New Solution: The MP-ViT (Multi-Plane Vision Transformer)
The authors of this paper built a new AI model called MP-ViT. Think of this model as a team of two expert detectives working together, rather than one detective trying to do everything alone.

Two Specialized Eyes: Instead of forcing the images to change shape, the MP-ViT has two separate "brains" (encoders).
- One brain is an expert at looking at Axial (top-down) slices.
- The other brain is an expert at looking at Sagittal (side-view) slices.
- They look at the images exactly as they are, without squishing or stretching them. No information is lost.
The "Handshake" (Cross-Attention): After each detective does their own analysis, they don't just shout out their conclusion. They have a special meeting called "cross-attention."
- The Axial expert says, "I see something suspicious here."
- The Sagittal expert says, "Oh, looking at that same spot from the side, I can confirm it."
- They combine their notes to make a much smarter, more confident decision than either could alone.
The "Missing Piece" Clue (Modality Vector): Sometimes, a patient's scan is incomplete. Maybe they have the top view but forgot the side view, or maybe a specific type of contrast dye is missing.
- To handle this, the model uses a special "ID card" (a modality indication vector). It's like a checklist the model holds: "We have the Top View (Check), but no Side View (Empty)."
- This tells the AI, "Hey, you're missing a piece of the puzzle, so don't panic. Just use what you have and adjust your thinking accordingly." This makes the model very robust, even when the data is messy or incomplete.

The Results: Who Won the Race?
The researchers tested this new team (MP-ViT) against older, single-brain models (like standard Vision Transformers and CNNs) using a massive dataset of over 12,000 patients.

The Score: The MP-ViT won the race. It was significantly better at spotting hemorrhages than the other models.
The Numbers: It improved the accuracy (measured by a score called AUC) by about 5.5% compared to the standard Vision Transformer and 1.8% compared to the best traditional AI models.
The Proof: Even when the researchers removed the "Missing Piece Clue" (the modality vector), the model still did well, but adding that clue made it even better. This proves that the model is smart enough to handle real-world messiness.

Why This Matters (According to the Paper)
The paper emphasizes that this approach is crucial because real-world hospital data is rarely perfect. Scanners from different manufacturers produce images in different shapes and sizes, and doctors often scan patients in different orientations.

By letting the AI look at the images in their natural "shape" (top or side) and letting those views talk to each other, the MP-ViT avoids the "smashing the vase" problem. It keeps all the critical details intact, leading to a more reliable tool for detecting brain bleeds, especially when the data comes in various forms or is missing a few pieces.

In a Nutshell:
The paper introduces a smarter AI that doesn't force medical images to change shape. Instead, it uses two specialized "eyes" to look at different angles simultaneously and a special "checklist" to handle missing data, resulting in a much more accurate way to find brain hemorrhages.

Technical Summary: Multi-Plane Vision Transformer for Hemorrhage Classification

Problem Statement
The identification of intracranial hemorrhage (ICH) via Magnetic Resonance Imaging (MRI) is critical for diagnosis, particularly for distinguishing acute from chronic conditions and avoiding ionizing radiation exposure associated with CT. However, clinical MRI acquisitions often present significant challenges for deep learning models:

Anisotropic and Multi-Plane Data: To speed up imaging, scans are often acquired with thick slices, resulting in anisotropic volumes. Clinicians frequently utilize multiple planes (axial, sagittal, coronal) to visualize anatomical structures comprehensively, as certain hemorrhages are more visible in specific orientations.
Information Loss in Resampling: Traditional deep learning approaches (CNNs and standard Vision Transformers) typically require input images to be resampled to a fixed, isotropic plane (e.g., converting sagittal to axial). This process involves extensive down-sampling or up-sampling, leading to significant information loss and potential interpolation artifacts.
Inconsistent Modalities: Real-world clinical datasets often suffer from missing contrast information (e.g., a patient may have axial T2 FLAIR but lack sagittal T1), which standard models struggle to handle without performance degradation.

Methodology
The authors propose the Multi-Plane Vision Transformer (MP-ViT), a novel architecture designed to process axial and sagittal MRI data simultaneously without resampling or changing image orientations. The architecture is built upon the Vision Transformer (ViT) framework and consists of four primary modules:

Dual-Branch Encoders: The model employs two separate transformer encoders: one dedicated to axial inputs and another to sagittal inputs.
- Input images are resized to fixed anisotropic dimensions (e.g., $256 \times 256 \times 32$ for axial and $256 \times 32 \times 256$ for sagittal) rather than being forced into a common isotropic grid.
- Images are divided into non-overlapping 3D patches ( $16 \times 16 \times 16$ ), tokenized, and processed by their respective encoders.
- This design treats axial and sagittal images as distinct modalities, preserving the native resolution and spatial relationships of each plane.
Cross-Attention Fusion: To integrate information across the two branches, the model utilizes a cross-attention mechanism. Specifically, the Classification (CLS) token from the axial encoder attends to the patch tokens of the sagittal encoder (and vice versa). This allows the model to exchange abstract information between planes more efficiently than standard all-attention fusion.
Modality Indication Vector: To address the issue of missing contrasts (e.g., absent sagittal T1 or specific axial sequences), the authors introduce a modality indication vector.
- This is a binary vector representing the presence (1) or absence (0) of specific contrasts available for each branch.
- A Multi-Layer Perceptron (MLP) embeds this vector into query vectors, which then perform cross-attention with the transformer output. This guides the model to compensate for missing data, enhancing robustness in heterogeneous clinical settings.
Classification Heads: The final output is derived from the mean of two classification heads, one for each branch, providing the final hemorrhage classification.

Key Contributions

Orientation-Agnostic Processing: The MP-ViT eliminates the need to resample anisotropic images to a fixed plane, thereby preventing the information loss and distortion associated with traditional interpolation methods.
Cross-Plane Integration: By utilizing cross-attention between separate axial and sagittal encoders, the model effectively fuses multi-planar information without architectural constraints forcing a single orientation.
Robustness to Missing Data: The introduction of the modality indication vector allows the model to adapt to incomplete input data (missing contrasts) without requiring imputation or data augmentation, a common challenge in retrospective clinical datasets.

Experimental Results
The model was evaluated on a real-world clinical dataset comprising 10,084 training, 1,289 validation, and 1,496 test subjects from three scanner manufacturers (GE, Philips, Siemens). The dataset included various contrasts (FLAIR, ADC, Trace, GRE, SWI, T2w, and sagittal T1w).

Performance: The MP-ViT-base variant achieved an Area Under the Curve (AUC) of 0.854.
Comparisons: This result outperformed:
- Standard Vision Transformer (ViT-base) by 5.5% (AUC 0.799).
- ResNet152 by 3.7% (AUC 0.817).
- DenseNet201 by 1.8% (AUC 0.836).
- Other transformer-based approaches (Multi-stage Transformer and Adel et al.'s ICH classifier) by 4.6% and 3.2%, respectively.
Statistical Significance: McNemar's tests confirmed that the improvements of MP-ViT over ViT, ResNet, DenseNet, and other transformer baselines were statistically significant ( $p < 0.05$ ).
Ablation Study: Removing the modality indication vector resulted in a performance drop (AUC 0.842 vs. 0.854), confirming the vector's role in handling missing contrasts.
Efficiency: Notably, MP-ViT-small (with fewer parameters than ViT-base) outperformed ViT-base by 1.5%, suggesting that the multi-plane architecture is more effective than simply increasing model depth or width.

Significance and Claims
The paper claims that MP-ViT offers a significant advancement in handling the variability of real-world clinical MRI data. By avoiding the resampling of anisotropic images, the model retains critical diagnostic information that might otherwise be lost. The authors emphasize that while MRI is not always the first choice for emergent ICH detection (due to scan time), it is crucial for stroke differentiation and non-urgent diagnostic workups where incidental hemorrhages are found.

The proposed method is presented as a mechanism to improve classification performance in scenarios requiring different orientation contrasts. The authors note that the approach is particularly relevant for clinical settings where data completeness varies, as the modality vector ensures the model remains robust even with incomplete inputs. The study concludes that these mechanisms should be considered for system designs dealing with the inherent variability of clinical data, though the authors acknowledge limitations, including the exclusion of coronal planes (due to dataset unavailability) and the focus on classification rather than segmentation.

Technical Summary: Multi-Plane Vision Transformer for Hemorrhage Classification

More like this