Leakage-Free Evaluation of Machine-Learning Models for Alzheimer’s Disease Classification from Handwriting Dynamics

Technical Summary: Leakage-Free Evaluation of Machine-Learning Models for Alzheimer's Disease Classification from Handwriting Dynamics

Problem Statement
Alzheimer's disease (AD) is a progressive neurological disorder requiring early identification for effective clinical management. While handwriting analysis has emerged as a promising non-invasive digital biomarker due to its reliance on memory, attention, and motor coordination, existing machine-learning studies utilizing the DARWIN handwriting dataset have reported highly inconsistent performance metrics. Reported accuracies for the same dataset range from 80.18% to 99.3%. This discrepancy suggests significant methodological flaws in previous studies, including potential information leakage, overfitting, inconsistent preprocessing, and validation designs that fail to separate model optimization from unbiased evaluation. The core problem addressed is the lack of a rigorous, leakage-free framework to determine the realistic predictive capability of handwriting-derived characteristics for AD classification.

Methodology
The study proposes a leakage-free machine-learning framework utilizing the DARWIN dataset, which comprises 174 participants (89 with AD, 85 healthy controls) and 450 numerical handwriting features (temporal, kinematic, pressure-related, and geometric).

• Validation Strategy: The authors implemented a repeated nested stratified cross-validation framework.
  • Outer Loop: Ten repetitions of stratified five-fold cross-validation, generating 50 independent outer-test evaluations.
  • Inner Loop: Stratified five-fold cross-validation used exclusively for hyperparameter optimization and feature selection.
• Leakage Prevention: Strict isolation was maintained where all data-dependent procedures—including median imputation, standardization, SelectKBest feature selection, optional SMOTE (Synthetic Minority Over-sampling Technique), and hyperparameter tuning—were performed only within the training folds of the inner loop. The outer-test folds remained completely untouched until the final evaluation.
• Models Evaluated: Four classifiers were compared under identical conditions: Logistic Regression, Support Vector Machine (SVM), Random Forest, and XGBoost.
• Hyperparameter Optimization: Randomized search with 30 candidate configurations per classifier, optimizing for ROC-AUC.
• Evaluation Metrics: Performance was assessed using Accuracy, Balanced Accuracy, Precision, Sensitivity, Specificity, F1-score, Matthews Correlation Coefficient (MCC), ROC-AUC, and Brier score.
• Explainability: Post-hoc analysis included SHAP (SHapley Additive exPlanations) and permutation importance on a model re-fitted to the full dataset for descriptive purposes, alongside a stability analysis of feature selection across folds.

Key Results

• Performance: Random Forest achieved the highest mean performance across the 50 outer-test evaluations:
  • Accuracy: 88.11%
  • Balanced Accuracy: 88.10%
  • F1-score: 88.43%
  • MCC: 0.7675
  • ROC-AUC: 0.9637
• XGBoost: Achieved the second-highest discrimination performance (Accuracy: 86.72%, ROC-AUC: 0.9501) but recorded the lowest Brier score (0.0936), indicating the lowest average probabilistic prediction error among the models.
• Variability: Random Forest demonstrated the most consistent performance across folds, while SVM showed greater variability.
• Feature Stability: The XGBoost feature-selection stability analysis revealed that the optimization procedure retained all 450 features in every outer fold (100% selection frequency), suggesting the model relies on the full feature set rather than a compact subset.
• Explainability: SHAP analysis indicated that predictive information was distributed across multiple correlated variables rather than dominated by a single feature. Permutation importance values were near-zero, further suggesting high feature redundancy.

Significance and Claims
The paper claims its primary contribution is the development of a methodologically controlled, leakage-free framework that provides a realistic internal estimate of AD classification performance using handwriting dynamics.

• Resolution of Inconsistency: The study's Random Forest result (88.11%) sits between the previously reported extremes of 80.18% and 99.3% for the same dataset. The authors argue that the lower result compared to the 99.3% claim is not a weakness but a reflection of rigorous validation that prevents information leakage and over-optimistic estimates.
• Reliability: By averaging performance over 50 unseen outer-test evaluations with strict separation of training and testing data, the study offers a more reliable assessment of generalization potential than single-split or non-nested cross-validation methods.
• Clinical Context: The authors emphasize that while handwriting analysis is a valuable, non-invasive screening tool, the current framework represents a supportive approach rather than a definitive diagnostic replacement. They explicitly state that external validation on larger, demographically diverse datasets is necessary before clinical deployment.
• Modesty: The paper modestly acknowledges limitations, including the small sample size (174 participants), the lack of external validation, the correlation between repeated fold scores, and the fact that feature importance analyses were descriptive rather than causal. It concludes that the findings support the utility of handwriting characteristics but do not yet establish specific biomarkers or clinical thresholds.