📄 medicine

A Leakage-Aware Machine Learning Framework for Multiclass Comorbidity Classification in Fibromyalgia: Integrating Genetic Feature Selection, Nested Cross-Validation, and Explainable AI

This paper presents a leakage-aware machine learning framework that integrates genetic feature selection, rigorous nested cross-validation with fold-specific preprocessing, and explainable AI to achieve high-accuracy multiclass classification of fibromyalgia comorbidities, identifying a weighted soft voting ensemble as the top performer and key clinical predictors via SHAP and LIME.

Original authors: Asmaa A. Attwa, Ahmed Yehia Ismaeel, Ahmed A. Elngar

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Asmaa A. Attwa, Ahmed Yehia Ismaeel, Ahmed A. Elngar

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the suspects are wearing identical masks. In the world of medicine, this happens when patients suffer from different conditions that share the same confusing symptoms. Fibromyalgia is a chronic condition causing widespread pain and fatigue, but it often shows up alongside other issues like a slipped disc in the back (Disc Herniation) or a sensitive, inflamed gut (Inflammatory Bowel Syndrome). Because these conditions all make people feel tired, achy, and generally unwell, it's incredibly hard for doctors to tell them apart just by looking at a patient's chart. This is where machine learning steps in. Think of machine learning as a super-smart computer brain that can spot tiny, hidden patterns in mountains of data that human eyes might miss. Instead of just guessing, the computer learns from past cases to predict what a new patient is likely suffering from. But there's a catch: if you teach the computer using the answers before the test, it will just memorize the answers and fail in the real world. This paper is about building a computer detective that is smart enough to learn the right way, without data leakage, and then explaining why it made its choices.

The researchers in this paper tackled the tricky problem of sorting fibromyalgia patients into three groups: those with no extra health issues, those with a slipped disc, and those with inflammatory bowel syndrome. They used a dataset of 59 patients, which is a very small group for a computer to learn from, making the task like trying to teach a dog to recognize three different breeds when you only have five examples of each. To solve this, they built a "leakage-aware" framework. Imagine you are training a student for a final exam. A "leaky" framework would be like letting the student peek at the answer key while studying, so they get a perfect score but fail the real test. This team made sure the student (the computer model) never saw the answers until the very end. They used a clever two-step process: first, they picked the most important clues (features) from a huge list of 242 medical measurements, narrowing it down to just 24 key clues. Then, they trained their computer models using a strict "nested" cross-validation method. This is like giving the student a series of practice tests where the questions change every time, ensuring they actually learn the material rather than just memorizing the practice answers.

After training sixteen different types of computer brains, the team found that a "Weighted Soft Voting" ensemble was the champion. Think of this as a team of five different experts (like a doctor, a nurse, a specialist, and two researchers) who all vote on the diagnosis. Instead of a simple majority vote, the team leader listens more closely to the experts who have proven to be more accurate in the past. This team approach achieved an accuracy of 0.864, meaning it correctly identified the condition in about 86 out of 100 cases. While a single computer model called "Extra Trees (Tuned)" was the best solo performer with an accuracy of 0.848, the team effort was slightly better. Interestingly, the researchers also found that a model called "Gradient Boosting" was the most reliable when it came to giving probability estimates. It was the most honest about its confidence, making it the best choice if a doctor needs to know exactly how likely a diagnosis is before making a treatment decision.

The paper didn't just stop at getting the right answer; it also asked the computer to explain its reasoning. Using tools called SHAP and LIME, the researchers peeked inside the computer's "brain" to see which symptoms mattered most. They discovered that the computer relied heavily on a specific question about symptom severity (SSS item 13), which asks about things like headaches, bowel issues, and fatigue. This makes perfect sense because the severity and variety of these symptoms are what usually distinguish the different conditions. For patients with a slipped disc, the computer also paid attention to cardiovascular symptoms and pain levels. For those with bowel issues, it looked at gut-related symptoms. The computer's logic matched what real doctors know about how these diseases work, giving the researchers confidence that the model wasn't just guessing randomly.

However, the authors are very careful not to overhype their results. They explicitly admit that while their method for training the models was strict and fair, the step where they picked the 24 best clues was done using the entire group of patients at once, rather than being re-tested inside every single practice round. They call this a "leakage" in the feature selection stage, which means the accuracy numbers they reported (like the 0.864 score) might be slightly too optimistic. It's like if the student picked their study guide based on the whole class's answers before the practice tests started. While the ranking of which computer model was best is likely correct because they all used the same clues, the exact score might drop a bit if tested on a completely new, larger group of people. The study concludes that while this framework is a powerful and transparent tool for understanding these complex conditions, it needs to be tested on much larger groups of patients from different hospitals before it can be used to make real-life medical decisions. The authors suggest that future work should focus on expanding the list of conditions the computer can detect and testing it in the real world to ensure it works as well outside the lab.

Technical Summary: A Leakage-Aware Machine Learning Framework for Multiclass Comorbidity Classification in Fibromyalgia

1. Problem Formulation

The study addresses the clinical challenge of differentiating between three distinct comorbidity states in patients with Fibromyalgia (FM): No Comorbidity (NC), Disc Herniation (DH), and Inflammatory Bowel Syndrome (IBS). These conditions share overlapping symptom profiles (e.g., widespread pain, fatigue, autonomic dysregulation), making differential diagnosis difficult using standard clinical assessments.

The problem is framed as a multiclass classification task ( $K=3$ ) using routine, multi-instrument clinical data. The dataset, Clinical_fm_66, initially contains 66 participants with 562 raw features derived from psychometric, neurological, and sociodemographic instruments (e.g., HAM-A, FIQ-R, WPI, SSS). After clinician-directed pruning and filtering of transitional labels, the analytical cohort consists of 59 patients with 242 features. The data presents significant challenges: high dimensionality relative to sample size, severe class imbalance (NC: 67.8%, DH: 23.7%, IBS: 8.5%), and missing values.

2. Methodology

The authors propose a "leakage-aware" pipeline that rigorously separates preprocessing stages to prevent information leakage from the test set, while explicitly acknowledging where full-cohort exposure occurs.

Data Preprocessing and Imputation:

Global Imputation: A $k$ -nearest-neighbor (KNN, $k=2$ ) pass is applied once to the full analytical cohort to resolve missing values before any cross-validation split.
Fold-Specific Imputation: A Multivariate Imputation by Chained Equations (MICE) imputer is nominally defined to be refit within each outer training fold, though the authors note that because the global KNN pass already eliminated missing values, this step functionally operates on complete data during the benchmark.
Scaling: $z$ -score standardization is strictly refit on the training partition of each outer fold and applied to the held-out fold.

Feature Selection (Two-Stage):

Stage 1 (Global): Mutual Information (MI) ranking is computed on the full cohort to reduce 242 features to the top 50.
Stage 2 (Global): A Genetic Algorithm (GA) with a population of 40 over 25 generations refines the 50 MI-selected features down to a 24-feature subset. The fitness function uses a 3-fold inner cross-validation with a Random Forest classifier.
Note: The authors explicitly state that both feature selection stages operate on the full cohort prior to the outer cross-validation split, a design choice that introduces potential optimistic bias in absolute performance metrics but is disclosed transparently.

Class Imbalance Handling:

BorderlineSMOTE is the primary augmentation strategy, applied strictly within the training partition of each outer fold. It generates synthetic samples for minority classes only near the decision boundary.
A fallback to standard SMOTE is used if BorderlineSMOTE fails to fit a stable boundary.
Validation folds retain the original, imbalanced distribution to ensure metrics reflect performance on realistic data.

Model Benchmarking and Optimization:

Classifiers: Sixteen supervised classifiers across six algorithmic families were evaluated: Random Forest, Extra Trees, Gradient Boosting, LightGBM, SVM, Bagging, and various meta-ensembles (Soft Voting, Weighted Voting, Stacking).
Validation: A 5-fold stratified nested cross-validation framework was used.
- Outer Loop: 5 folds for model evaluation.
- Inner Loop: 3 folds for hyperparameter tuning (using Optuna with Tree-structured Parzen Estimator) and feature selection fitness evaluation.
Hyperparameter Optimization: Bayesian optimization was applied to five base classifiers (RF, ET, GB, LightGBM, SVM) with 50 trials per model.
Ensembling: Top-performing models were aggregated via Soft Voting, Weighted Soft Voting (weights optimized via Optuna), and Stacking (using Logistic Regression and Random Forest meta-learners).

Explainability and Calibration:

SHAP (TreeExplainer): Used for global and per-class feature importance analysis.
LIME: Used for local instance-level explanations.
Calibration: Expected Calibration Error (ECE) and Brier scores were computed per fold to assess probability reliability.

3. Key Results

Predictive Performance: The Weighted Soft Voting ensemble achieved the highest discriminative performance with a mean accuracy of 0.864, an F1-score of 0.826, and a mean nested-CV macro-AUC of 0.866.
Single Classifier Performance: Extra Trees (Tuned) was the strongest individual classifier (Accuracy: 0.848, F1: 0.825).
Calibration: Gradient Boosting demonstrated the best probability calibration (ECE = 0.187 ± 0.016, Brier Score = 0.103), making it a strong candidate for threshold-based clinical decision support despite lower point-estimate accuracy.
Statistical Significance: The Friedman omnibus test confirmed significant differences across classifiers ( $\chi^2 = 42.25, p < 0.001$ ). Post-hoc Wilcoxon signed-rank tests with Bonferroni correction showed that ensemble methods significantly outperformed weaker base models (e.g., Weighted Voting vs. SVM (Tuned) had a Cohen's $d$ of 2.14).
Feature Importance: SHAP analysis identified Symptom Severity Scale (SSS) item 13 as the dominant predictive signal globally.
- No Comorbidity: Distinguished by lower symptom severity (SSS item 18).
- Disc Herniation: Characterized by SSS item 13, SSS item 1, and cardiovascular symptoms.
- IBS: Characterized by SSS item 13, SSS item 6 (bowel symptoms), and physical function subscales, consistent with gut-brain axis dysregulation.
Inference Latency: Gradient Boosting and LightGBM variants offered the lowest inference times (<0.25 ms/sample), while meta-ensembles required significantly more time (>100 ms/sample).

4. Key Contributions

Multiclass Framework: A novel three-class supervised learning framework distinguishing NC, DH, and IBS in Fibromyalgia, moving beyond binary disease detection.
Leakage-Aware Pipeline: A preprocessing protocol that strictly isolates imputation, scaling, and oversampling within training folds. The authors explicitly disclose that feature selection (MI and GA) was performed on the full cohort, distinguishing this "leakage-free classifier training" from a fully leakage-free feature selection process.
Comprehensive Benchmarking: A rigorous evaluation of 16 classifiers under nested cross-validation with global statistical validation (Friedman, Wilcoxon) and effect size analysis.
Explainability Integration: Systematic application of SHAP and LIME to recover clinically plausible pathophysiological signatures for each comorbidity class, linking model attributions to known medical mechanisms (e.g., visceral hypersensitivity in IBS).
Calibration Focus: Explicit evaluation of probability calibration (ECE, Brier score) alongside accuracy, addressing a gap in clinical ML literature where decision-support tools require reliable probability estimates.

5. Significance and Limitations (as claimed by the paper)

The paper claims its primary significance lies in providing a methodologically transparent and reproducible framework for multiclass comorbidity classification that avoids common pitfalls like data leakage in classifier training. It demonstrates that ensemble methods can achieve strong discriminative performance even in small-sample, high-imbalance clinical settings when rigorous cross-validation and appropriate resampling (BorderlineSMOTE) are used.

However, the authors maintain a modest stance regarding the absolute performance figures:

Optimistic Bias: They explicitly acknowledge that because feature selection (MI and GA) was performed on the full cohort rather than nested within the outer folds, the reported accuracy and F1 scores are likely optimistic estimates of generalization performance. The relative ranking of models is considered more robust, as all models were exposed to the same feature subset.
Generalizability: The study is limited by the small sample size ( $n=59$ ) and single-site data collection. The authors state that external validation on independent, multi-center cohorts is a prerequisite for clinical deployment.
Causal Claims: The authors clarify that SHAP and LIME provide evidence of construct validity (model consistency with clinical knowledge) rather than causal proof of disease etiology.

The work concludes that while the current pipeline offers a strong starting point for clinical decision support, future work must prioritize fully nested feature selection, external validation, and the expansion of the label space to include psychiatric and metabolic comorbidities.