MedicalRec: Medical recommender system for image classification without retraining

Technical Summary: MedicalRec: Medical Recommender System for Image Classification Without Retraining

Problem Statement
The rapid adoption of deep learning (DL) in medical image classification has introduced significant environmental and computational challenges. Training state-of-the-art models requires substantial energy, leading to high carbon emissions and electronic waste. A primary inefficiency in current workflows is the "trial and error" process researchers and practitioners undergo to select the optimal model for a specific medical image dataset. This process involves training and evaluating dozens of models to identify the best performer, a practice that exacerbates carbon footprints and consumes excessive time and resources. Furthermore, the lack of a standardized benchmark or protocol for recommending models based on dataset characteristics forces this repetitive and resource-intensive cycle.

Methodology
To address these challenges, the authors propose MedicalRec, a Transformer-based recommender system designed to predict the most suitable machine learning model for a given medical image classification task without requiring retraining.

1. Dataset Construction (MedicalRec-Bench):
   The foundation of the study is a new dataset, MedicalRec-Bench, compiled from 3,000 scientific articles in the field of medical image classification. This dataset contains over 5,000 records of models tested across various tasks, including Skin Cancer, Tumour, Wound, Breast Cancer, and MRI classification.

   • Features: The dataset includes metadata such as titles, abstracts, keywords, and statistical features (e.g., sample size, image dimensions, number of classes) alongside performance metrics (Accuracy, Precision, Recall, F1, AUC).
   • Versions: Due to the challenge of non-reporting by original authors (resulting in significant missing values), the dataset was released in four versions with varying feature counts: MedicalRec I (5 features), II (9 features), III (11 features), and IV (18 features).
   • Data Preprocessing: Missing values (represented as 0.0) were imputed using column averages. Textual data (Model names) was embedded using BERT, while numerical features were used directly.

2. Model Architecture:
   MedicalRec is formulated as a sequential recommendation problem where:

   • User/Scenario: A specific classification task defined by dataset characteristics (e.g., image size, number of classes).
   • Item: A specific model architecture (e.g., ResNet-50, VGG-16).
   • Score: The performance metric (primarily Accuracy).

   The model utilizes a Transformer architecture inspired by BERT4Rec. It employs:

   • Embedding Layer: Converts input features into continuous vectors.
   • Transformer Encoder: Consists of 12 layers, each with 12 self-attention heads, to capture dependencies between input features and model performance.
   • Positional Embeddings: Added to preserve temporal/sequential information.
   • Output Layer: A Softmax function ranks potential model recommendations probabilistically.
   • Training Objective: The model is trained using a masked item prediction task (Cloze task), where a proportion of input items are masked, and the model predicts them using a two-layer feed-forward network with GELU activation.

Key Contributions

• MedicalRec-Bench: The creation of the first publicly available dataset specifically designed for model recommendation in medical image classification, aggregating data from 3,000 articles.
• MedicalRec Model: The development of a Transformer-based recommender system that maps dataset characteristics to optimal model architectures, eliminating the need for exhaustive retraining.
• Comprehensive Evaluation: A rigorous evaluation of the proposed model against 12 baseline models (including Collaborative Filtering, Content-Based, SVD, LSTM, GRU, IndRNN, LLM, and Fairness-aware models) across four dataset configurations.

Results
The study evaluated the models using Precision@K, Recall@K, F1@K, Discounted Cumulative Gain (DCG), and HitRate@K.

• Performance: The proposed MedicalRec model achieved superior performance across all dataset versions (I–IV). In the most comprehensive configuration (MedicalRec IV), it achieved a HitRate@100 of 75.50%, a Recall of 9.80%, and a Precision of 10.50%.
• Comparison: MedicalRec consistently outperformed classical methods (Collaborative Filtering, SVD) and deep learning baselines (LSTM, GRU, IndRNN). While "Fair" and "Multi-modal" models showed competitive results in specific metrics (e.g., DCG@5), MedicalRec maintained the highest overall scores, particularly in longer recommendation lists (D@50 to D@100).
• Ablation Studies:
  • Activation Functions: ReLU was found to yield the highest F1@100 scores across all dataset versions.
  • Dropout: Optimal dropout rates varied by dataset complexity (0.7 for MedicalRec I; 0.5 for II, III, and IV).
• Statistical Significance: K-fold cross-validation (5-fold) on MedicalRec IV confirmed the model's stability, with HitRate@100 ranging between 75.22% and 76.50%. T-tests (P-values) indicated that the performance difference between MedicalRec and all other models was statistically significant (P < 0.05).

Significance and Claims
The paper claims that MedicalRec offers a practical solution to reduce the environmental impact of medical AI development. By recommending the optimal model based on dataset features, the system aims to eliminate the need for researchers to train multiple candidate models, thereby saving energy and reducing carbon emissions. The authors emphasize that this is the first work to utilize statistical features from existing literature to build a model recommendation system for medical imaging. The dataset and implementation are made publicly available to facilitate future research, with the authors noting that future work could address challenges such as zero-starting values, missing data imputation, and class imbalance.