🤖 machine learning

MS-MLB: An Open Machine Learning Benchmark for Blood-Based MS Classification

This paper introduces MS-MLB, the first open and reproducible machine learning benchmark for classifying Multiple Sclerosis versus healthy controls using whole blood RNA expression data from the GSE17048 cohort, featuring a rigorous, leakage-controlled evaluation pipeline that identified Gradient Boosting as the top-performing algorithm.

Original authors: Adam Simson, Ankush Dutta, Quang Bui

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Adam Simson, Ankush Dutta, Quang Bui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the human body as a bustling city. Inside this city, there's a security force called the immune system, whose job is to patrol the streets and keep things safe. Sometimes, however, this security force gets confused and starts attacking the city's own buildings—in this case, the central nervous system. This mix-up is called Multiple Sclerosis (MS). To figure out if someone has MS, doctors usually act like detectives: they look at the patient's symptoms, scan their brain with a giant magnet (MRI), and check for other possible reasons for the trouble. But what if there was a secret message hidden in the blood? Blood is like a river flowing through the city, carrying tiny messengers (RNA) that tell stories about what the immune system is doing. Scientists have wondered if we could read these stories to spot MS early, but it's tricky. The "river" is full of noise, and the stories are written in a language with thousands of words, making it hard to find the right ones without getting confused.

This is where a new project called MS-MLB steps in. Think of it not as a magic crystal ball that instantly diagnoses patients, but as a giant, transparent playground for computer scientists. The researchers built a fair, open "game" where different computer programs (algorithms) can try to guess who has MS and who is healthy, based on those blood stories. The goal wasn't to create a final medical tool for doctors to use tomorrow, but to build a strict, unchangeable rulebook so that everyone plays by the same rules. This way, we can finally see which computer program is actually the best at spotting the signal in the noise, without anyone peeking at the answers before the game starts.

The Big Game: MS-MLB

The paper introduces MS-MLB (Multiple Sclerosis – Machine Learning Benchmark), a reproducible open benchmark designed to test how well computers can classify MS using whole blood RNA data. The researchers took a public dataset called GSE17048, which contains blood samples from 144 individuals: 99 people with MS and 45 healthy controls. They turned this into a simple "MS vs. Healthy" guessing game.

The biggest challenge in this game is that the "story" (the data) has thousands of words (genes), but there are only a few players (samples). If you aren't careful, a computer might memorize the answers instead of learning the pattern. To stop this, the MS-MLB framework acts like a strict referee. It forces every computer program to follow a specific pipeline:

Clean the data only using the training group.
Pick the best words (features) only from the training group.
Train the model on the training group.
Test the model only on the "untouched" group it has never seen before.

This prevents "leakage," where information from the test group accidentally sneaks into the training phase, which would make the computer look smarter than it really is.

The Results: Who Won the Game?

The researchers ran the game with several different types of computer brains, including Logistic Regression, Support Vector Machines, Random Forests, and Gradient Boosting. They used a special scoring system called the MS Research Score to rank them. This score isn't just about being right; it weighs how well the model distinguishes between the two groups (AUC-ROC), how many sick people it catches (Sensitivity), how many healthy people it correctly leaves alone (Specificity), and how confident it is in its guesses (Calibration).

Here is what happened on the final "untouched" test set of 29 samples:

The Champion: Gradient Boosting took first place with an MS Research Score of 93.83.
- It had an AUC-ROC of 0.989 (almost perfect at ranking).
- It caught 95.0% of the MS cases (Sensitivity).
- It correctly identified 77.8% of the healthy people (Specificity).
- Its Brier score was 0.050, meaning its probability guesses were very close to the truth.
The Runner-up: Neural Network came in second with a score of 91.24.
Third Place: Random Forest scored 90.47. Interestingly, Random Forest caught 100% of the MS cases but mislabeled more healthy people, showing a different kind of "error" than the others.

The paper also looked at how the models performed during the training phase using nested cross-validation. In this stricter test, SVM actually scored the highest (77.62), followed by Neural Network and Logistic Regression. This difference shows that a model's ranking can change depending on how you slice the data, which is why the authors emphasize looking at the whole picture, not just one number.

What This Means (and What It Doesn't)

The paper suggests that there is a learnable signal in the blood RNA data that can distinguish MS from healthy controls under these strict rules. The high scores suggest the computer programs found real patterns. However, the authors are very careful to say what this is not:

It is not a medical diagnosis tool. The score is called the "MS Research Score" for a reason. It is for comparing research models, not for telling a real patient they have MS.
It is not a cure or a final answer. The dataset is small (only 144 people), and the "untouched" test set was tiny (only 29 people). A few different guesses could change the results significantly.
It does not prove which genes are the "culprits." While the models picked certain genes to make their guesses, the paper does not claim these genes are the biological cause of MS. They might just be reacting to medication, age, or other factors.

The true novelty of this paper isn't the specific computer program that won; it's the playground itself. Before this, researchers might have built their own rules, making it hard to compare their results with others. MS-MLB provides a shared, open framework where anyone can submit their own model, run it through the same strict pipeline, and see how it stacks up against the baseline. It turns a messy collection of individual studies into a fair, transparent competition.

In short, the paper suggests that with the right rules, computers can spot MS signals in blood RNA with impressive accuracy. But it also reminds us that we are still in the research phase. The "champion" model is a great starting point for future scientists to build upon, but it's not ready to walk into a doctor's office just yet. The real victory here is the creation of a clear, reproducible path for everyone to follow, ensuring that future discoveries are built on solid ground rather than shaky guesses.

Technical Summary: MS-MLB – An Open Machine Learning Benchmark for Blood-Based MS Classification

Problem Statement
Multiple sclerosis (MS) is currently diagnosed through clinical assessment, MRI, and laboratory evidence, excluding other explanations. While blood RNA expression data contains immune signals relevant to MS, developing machine learning (ML) classifiers for this data faces significant reproducibility challenges. The primary technical hurdle is the "small $n$ , large $p$ " problem inherent in transcriptomics: datasets often contain thousands of gene features but few samples (e.g., 144 individuals). This ratio creates a high risk of data leakage, where information from validation or test sets inadvertently influences feature selection or model tuning, leading to inflated performance estimates. Furthermore, existing literature lacks a standardized, open framework that allows researchers to compare algorithms under identical, leakage-controlled conditions for the specific task of classifying MS versus healthy controls using whole blood RNA.

Methodology
The paper introduces MS-MLB (Multiple Sclerosis – Machine Learning Benchmark), a reproducible Python-based framework built on the public GSE17048 cohort (99 MS patients, 45 healthy controls). The methodology is designed to eliminate evaluation bias through a strict, shared pipeline:

Data Partitioning: The dataset is split into an 80% training set and a 20% untouched stratified holdout set (29 samples).
Leakage Control: The core technical innovation is the placement of all preprocessing steps inside a nested cross-validation pipeline. Specifically, supervised feature selection (SelectKBest), standard scaling, and SMOTE (Synthetic Minority Over-sampling Technique) are fitted only on the training fold of each iteration. This prevents validation labels from influencing feature selection or synthetic sample generation.
Evaluation Protocol:
- Nested Cross-Validation: A 5-fold outer loop with a 3-fold inner loop is used for hyperparameter tuning (via grid search targeting AUC-ROC) and performance estimation, separating model selection from performance assessment.
- Metrics: The benchmark reports accuracy, balanced accuracy, sensitivity, specificity, precision, F1, Matthews correlation coefficient, AUC-ROC, PR-AUC, and Brier score.
- Uncertainty: 2,000 bootstrap samples are used to calculate confidence intervals for AUC-ROC.
- Scoring: An exploratory MS Research Score (0–100) is calculated as a weighted sum of AUC-ROC (25%), PR-AUC (15%), sensitivity (25%), specificity (15%), F1 (10%), and calibration (10% via $1 - \text{Brier score}$ ).
Model Registry: The framework evaluates Logistic Regression, SVM, Random Forest, Gradient Boosting, KNN, Naive Bayes, Neural Networks, and a Stacking Ensemble.
External Submission: A documented interface allows external researchers to submit models (via Python dictionaries or pre-trained files) that are automatically wrapped in the same leakage-controlled pipeline for fair comparison.

Key Results
In the stored benchmark run on the holdout set:

Top Performer: Gradient Boosting achieved the highest MS Research Score of 93.83. It demonstrated an AUC-ROC of 0.989 (95% CI: [0.947, 1.000]), sensitivity of 0.950, specificity of 0.778, F1 score of 0.927, and a Brier score of 0.050.
Ranking: The Neural Network (91.24), Random Forest (90.47), Logistic Regression (90.26), and Stacking Ensemble (89.51) followed.
Trade-offs: The results highlight distinct error patterns; for instance, Random Forest achieved perfect sensitivity (1.000) but lower specificity (0.667), while KNN achieved perfect specificity (1.000) but poor sensitivity (0.550). The composite score was necessary to visualize these trade-offs.
Nested vs. Holdout: Nested cross-validation scores were generally more conservative than holdout scores (e.g., SVM scored 77.62 in nested CV vs. 87.01 on holdout), underscoring the importance of reporting both to avoid over-optimism.

Significance and Claims
The paper positions MS-MLB not as a clinical diagnostic tool, but as a reproducible infrastructure for research. Its significance lies in three specific claims:

First-of-its-Kind Benchmark: To the authors' knowledge, this is the first open benchmark specifically for GSE17048 whole blood RNA MS vs. healthy control classification that includes a documented external model submission pathway.
Reproducibility Standard: By fixing the preprocessing, leakage control, and scoring logic in a single entry point, the framework shifts the burden of correct evaluation from individual users to shared infrastructure. This allows researchers to focus on model innovation rather than reconstructing evaluation pipelines.
Modest Scope: The authors explicitly state that the "MS Research Score" is exploratory and not clinically validated. The benchmark detects learnable signals within the specific GSE17048 cohort but does not claim to identify validated biomarkers or replace clinical diagnosis. The task is strictly limited to pooled MS vs. healthy control classification, excluding subtype differentiation, clinically isolated syndrome (CIS), or treatment response.

Conclusion
MS-MLB provides a rigorous, leakage-controlled environment to compare ML algorithms on blood-based MS data. While Gradient Boosting currently leads the benchmark, the primary contribution is the framework itself, which enables fair comparison, external validation, and the identification of generalization gaps in high-dimensional biomedical machine learning. Future work is directed toward external validation on independent cohorts, feature stability analysis, and pathway mapping, rather than immediate clinical deployment.

The Big Game: MS-MLB

The Results: Who Won the Game?

What This Means (and What It Doesn't)

Technical Summary: MS-MLB – An Open Machine Learning Benchmark for Blood-Based MS Classification

More like this