Deep neural networks with Fisher vector encoding for medical image classification

Technical Summary: Deep Neural Networks with Fisher Vector Encoding for Medical Image Classification

Problem Statement
Convolutional Neural Networks (CNNs) have become the standard for image recognition but face limitations in medical image classification, including the requirement for fixed input sizes (often leading to information loss via pooling) and a "locality bias" that hinders the capture of global information. While Vision Transformers (ViT) address locality bias through self-attention, they typically require large datasets for training and exhibit quadratic computational complexity relative to image resolution, making them less suitable for high-resolution medical images or scenarios with limited data availability. Furthermore, while hybrid CNN+ViT models have emerged to mitigate these issues, the integration of sophisticated, orderless feature representation techniques—specifically Fisher Vectors (FV)—with these hybrid architectures remains largely unexplored. A significant barrier to applying Fisher Vectors in large-scale settings is the high computational cost of estimating the underlying Gaussian Mixture Model (GMM).

Methodology
The authors propose a hybrid architecture that combines MIT-EfficientViT (a multi-scale Vision Transformer) with Fisher Vector encoding to create a model suitable for both small and large medical datasets. The core methodology involves the following components:

1. Architecture Modification: The standard classification head of the MIT-EfficientViT model is replaced. Instead of a Mobile inverted Bottleneck Convolution (MBConv) layer followed by global pooling, the authors insert a Fisher Vector encoding layer.
2. Lossless Feature Concatenation: To address the issue of concatenating local features from multiple stages of the Transformer backbone (which exist in different dimensional spaces), the authors propose a lossless approach. Rather than downsampling higher-dimensional features (which causes information loss) or upsampling lower-dimensional ones (which increases computational cost), they calculate the Greatest Common Divisor (GCD) of the feature dimensions. Local features are split into sub-features of this common dimension and then concatenated.
3. GMM Estimation via Entropy-Based Subsampling: To overcome the computational bottleneck of GMM estimation on large datasets, the authors introduce a sampling strategy based on image entropy. They calculate the Shannon entropy of images in the training set and select a subset of samples (up to a limit, e.g., 5,000) with the highest entropy values to estimate the GMM parameters. This aims to retain the most informative samples while reducing computational load.
4. Adaptation for 3D Data: To handle 3D medical images (e.g., CT scans), the 2D convolutional layers of the backbone are replaced with 3D convolutions. Since pre-trained 3D weights are unavailable, the authors initialize the model by repeating 2D pre-trained weights across one axis.
5. Training Strategy: The backbone is fine-tuned on the target dataset before extracting features for Fisher Vector encoding. This step is crucial for generating application-specific high-level features, particularly for 3D data where no pre-trained weights exist.

Key Contributions
The paper outlines four primary contributions:

• Robust GMM Estimation: A method for estimating Gaussian Mixture Models by subsampling the training set based on image entropy. Experiments demonstrate this reduces the number of samples required for GMM training without statistically significant loss in model performance.
• Hybrid Medical Classification Model: The introduction of a medical image classification model combining hybrid CNN+ViT architectures with Fisher Vector encoding.
• Lossless Feature Concatenation: A novel, lossless approach for concatenating features from multiple Transformer stages, addressing information loss issues reported in prior work.
• Extension to 3D: The expansion of Fisher Vector applications to 3D image datasets by developing a methodology to retrieve knowledge from 2D pre-trained domains.

Results
The proposed method was evaluated on a diverse collection of datasets, including the MedMNIST (v2) suite (covering 2D and 3D modalities), Clean-CC-CCII (COVID-19 chest CTs), and ISIC2018 (dermatoscopic images).

• MedMNIST (v2): The model outperformed benchmark results (including ResNet variants, AutoML, and other hybrid models) across all MedMNIST datasets. Notable improvements of at least 5% in accuracy were observed in RetinaMNIST, OrganMNIST3D, FractureMNIST3D, and SynapseMNIST3D.
• Fine-tuning Impact: Fine-tuning the backbone prior to GMM estimation consistently improved both Accuracy (ACC) and Area Under ROC Curve (AUC) for both 2D and 3D cases. The improvement was particularly significant in 3D cases where pre-trained weights were absent.
• Feature Extraction Stages: Using features from two stages (Stage 4 and Stage 3) generally yielded optimal results, especially when the backbone was fine-tuned.
• Realistic Datasets: On Clean-CC-CCII and ISIC2018, the model achieved competitive results against recent literature (e.g., ResGANet, Hiera, MogaNet), outperforming them in Clean-CC-CCII and matching or exceeding them in ISIC2018, while maintaining a comparable parameter count (limited to ~25 million).
• Subsampling Efficacy: Analysis using Kullback-Leibler (KL) Divergence showed that a GMM estimated using only 2% of the training set (selected by entropy) was as similar to the base case (100% of data) as a GMM estimated using 50% of the data, validating the efficiency of the sampling strategy.

Significance and Claims
The authors claim that their work successfully bridges the gap between the data efficiency of orderless encoding methods (like Fisher Vectors) and the representational power of hybrid CNN+ViT models. By addressing the scalability issues of GMM estimation through entropy-based subsampling and solving feature concatenation issues via a lossless splitting method, the proposed model is presented as a robust solution for medical image classification across varying data scales and modalities. The results suggest that this approach is particularly well-suited for biomedical applications where data acquisition is expensive and datasets may be small or high-resolution. The authors modestly conclude that while their method is effective, future work could explore trainable first- and second-order statistics estimation to enable end-to-end learning and incremental learning capabilities.