Multimodal Brain Tumour Classification Using Feature Fusion

Technical Summary: Multimodal Brain Tumour Classification Using Feature Fusion

Problem Statement
Current deep learning approaches for brain tumor classification predominantly rely on unimodal inputs, utilizing only MRI or CT images. This limitation fails to replicate the clinical diagnostic workflow, where practitioners synthesize diverse information sources—including symptoms, medical history, and quantitative imaging data—into a unified judgment. The authors argue that relying solely on raw image data overlooks the potential benefits of integrating handcrafted radiomic features, which provide explicit quantitative descriptors of tumor characteristics often used in clinical reasoning.

Methodology
The study proposes a two-branch multimodal framework designed to classify brain tumors into four categories: glioma, meningioma, pituitary, and no-tumor. The architecture processes two distinct data streams:

1. Image Stream: Raw MRI scans (resized to 224×224) are processed through a pre-trained CNN backbone. Three architectures were evaluated: EfficientNet-B3, ResNet-50, and DenseNet-121. These backbones output a 256-dimensional visual embedding via a head consisting of dropout, linear, and Layer Normalization layers.
2. Radiomic Stream: A dedicated three-stage unsupervised segmentation pipeline extracts three spatial regions from each MRI: a brain mask, a tumor ROI, and a 15-pixel peritumoral ring. From these regions, 91 radiomic features are extracted across ten families, including first-order intensity statistics, shape descriptors, GLCM texture, GLRLM, NGTDM, GLDM, gradient/HOG features, and boundary/heterogeneity/fractal descriptors. These features are encoded by a three-layer MLP (Batch Norm, Linear layers, Layer Norm) into a matching 256-dimensional numerical embedding.

Feature Fusion Strategies
The two 256-dimensional embeddings are fused using one of three strategies to generate the final classification:

• Concatenation: Simple concatenation of the two vectors.
• Gated Fusion: Utilizes dual learned sigmoid gates to apply per-dimension modality weights.
• Bidirectional Cross-Modal Attention: Employs attention mechanisms with residual connections to allow the streams to interact.

Experimental Setup and Dataset
The framework was evaluated on a balanced dataset of 7,200 brain MRI images (1,600 per class for training, 400 per class for testing) compiled from multiple public repositories (Sartaj, BRaTS, etc.). The study acknowledges a limitation regarding the dataset: since patient identifiers were unavailable, the split is at the image level rather than the patient level, meaning potential patient overlap between training and testing sets cannot be excluded.

The experimental design consisted of a grid of nine runs (3 CNN backbones × 3 fusion strategies). Training involved freezing the backbone for 8 epochs followed by fine-tuning the last three blocks for 12 epochs, utilizing weighted sampling, label smoothing, gradient clipping, and mixed precision.

Key Results

• Unimodal Baselines: Among single-modality models, DenseNet-121 achieved the highest accuracy of 95.50%, followed by ResNet-50 (95.37%) and EfficientNet-B3 (94.94%). The meningioma class consistently presented the greatest classification challenge due to lower recall.
• Multimodal Performance: All multimodal configurations outperformed their corresponding unimodal baselines. The best performance was achieved by ResNet-50 + Gated Fusion and DenseNet-121 + Gated Fusion, both reaching an accuracy of 96.13%. Other strong performers included DenseNet-121 + Attention (96.06%) and DenseNet-121 + Concatenation (95.81%).
• Class-Specific Gains: The "no-tumor" and "pituitary" classes benefited significantly from multimodal fusion, achieving near-perfect recall. While meningioma remained the most difficult class, the gated fusion strategy demonstrated the most consistent gains across architectures.
• Comparison: The proposed framework (96.13%) is competitive with recent literature, such as Aziz et al. (2024), who reported 96.9% using a DenseNet-based MRI-only framework.

Significance and Claims
The paper claims that integrating handcrafted radiomic descriptors with deep CNN image embeddings via multimodal fusion provides a robust improvement over unimodal image-only baselines. The study highlights that gated fusion is the most effective strategy for combining these modalities.

However, the authors maintain a modest stance regarding the magnitude of these improvements. They explicitly acknowledge that the observed gains are relatively modest because both branches derive their information from the same MRI source. Consequently, the degree of true complementarity between the "modalities" is limited. The authors position this work as a preliminary step, noting that future research will focus on replacing the radiomic branch with independent clinical data sources (e.g., structured reports, lab results, patient history) to better replicate true multimodal clinical reasoning and introduce information entirely absent from the image domain.