A retrospective public external benchmark of healthy-to-stroke lower-limb EEG transport identifies constraints from source construction, adaptation burden, and confound sensitivity

This retrospective public benchmark demonstrates that healthy-to-stroke lower-limb EEG transport is currently weak and constrained more by source construction and adaptation burden than by model complexity, suggesting a need for harmonized prospective validation over further retrospective model iteration.

The Gist

Imagine you are trying to teach a robot how to walk. You have two groups of students:

1. The Healthy Students: They are fit, young, and can run and jump perfectly. You teach them how to move their legs using a special brain-reading headset.
2. The Stroke Patients: They have suffered a stroke, so their brains and legs don't work the same way as the healthy students. They need the robot to help them walk again.

The Big Question:
If you train your robot using only the Healthy Students, can you just hand that robot over to the Stroke Patients and expect it to work immediately? Or does the robot get confused because the "brain signals" of a stroke patient are totally different from a healthy person?

This paper is a big, honest report card on that exact question. The researchers tried to build a "universal translator" for brain signals, moving from healthy people to stroke patients, and here is what they found.

The Experiment: A "Test Drive" Across Different Roads

Think of the three datasets they used as three different driving schools:

• School A (EEGMMIDB): A huge school with healthy drivers doing various tasks.
• School B (MILimbEEG): Another school with healthy drivers, but they only practice specific leg movements.
• School C (Stroke2025): The target school. This is where the stroke patients are. This is the "final exam."

The researchers tried to take the driving skills learned in Schools A and B and apply them to School C without changing the car (the AI model).

The Results: The "Zero-Shot" Crash

"Zero-shot" means trying to drive the car from School A directly into School C without any practice or adjustments.

• The Result: It was a disaster. The robot was barely better than guessing. It was like trying to drive a Formula 1 car on a muddy farm road; the engine is too powerful and the tires are wrong.
• The Surprising Winner: The researchers expected the newest, fanciest AI (Deep Learning/EEGNet) to win. Instead, an older, simpler, "classic" method (CSP+LDA) performed slightly better. It was like an old, reliable sedan handling the mud better than the fancy sports car.

The "10-Shot" Fix: A Little Practice Helps, But Not Much

The researchers then tried "Few-Shot Learning." This is like giving the robot a crash course: "Here are 10 examples of how this specific stroke patient moves their leg. Now, try again."

• The Result: It helped a little, but not enough to be a miracle cure.
• The Twist: The robot didn't actually get smarter at understanding the signals (its "ranking" ability stayed the same). Instead, it just got better at calibrating its confidence.
  • Analogy: Imagine a weather forecaster who is always wrong. If you tell them, "Hey, when you say 50%, you're usually right," they might change their behavior. They won't suddenly become a genius meteorologist, but they will stop saying "It will definitely rain" when it's actually sunny. The robot learned to be less overconfident, which is useful, but it still couldn't perfectly predict the movement.

The "Source" Problem: Garbage In, Garbage Out

The researchers realized that where you get your training data matters more than how you build the AI.

• If they trained only on the "MILimbEEG" school data, the robot failed completely.
• If they mixed data from both healthy schools, it did slightly better.
• The Lesson: You can't just throw random healthy data at the problem. The "flavor" of the healthy data has to match the stroke data somewhat. But even with the best mix, the gap between healthy and stroke brains was too wide to bridge with just a little bit of data.

The "Motor" Mystery: Is it really the brain?

Finally, they checked: "Are we actually reading the motor part of the brain (the part that moves legs), or are we just picking up noise from the eyes or face muscles?"

• They tested different electrode setups (like changing the antenna on a radio).
• The Result: Even electrodes placed on the forehead (which shouldn't pick up leg signals) worked almost as well as the ones on the motor cortex.
• The Takeaway: This suggests the robot might be picking up general "effort" or noise rather than a pure, specific "I want to move my leg" signal from the brain. It's like hearing a crowd cheer and assuming it's a specific player scoring, when it might just be the crowd clapping.

The Bottom Line: What Should We Do Next?

The authors are saying: "Stop trying to build a better AI model. That's not the problem."

The problem is that we are trying to translate a language (healthy brains) into another language (stroke brains) without a dictionary, and the two languages are too different.

The Solution:
Instead of trying to fix the software (the AI), we need to fix the hardware and the experiment.

1. We need to record healthy people and stroke patients at the same time, using the same equipment, and asking them to do the exact same tasks.
2. We need to record other things too, like eye movements and muscle twitches, to make sure the robot isn't cheating.
3. We need to test this in the real world, not just on a computer.

In short: The robot isn't broken; the training manual is. We need a new, better manual that teaches the robot how to handle both healthy and stroke brains from day one, rather than hoping it can figure it out on the fly.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in neurorehabilitation and brain-computer interface (BCI) research: the lack of external validation for lower-limb EEG decoders.

• The Gap: Most existing lower-limb EEG studies are benchmarked within a single dataset (often healthy participants) and focus solely on discrimination performance. They fail to answer whether a decoder trained on healthy data can successfully transfer to a clinical stroke population.
• The Challenge: Translating from healthy "source" domains to stroke "target" domains involves significant domain shifts due to differences in acquisition protocols, task semantics, and physiological states.
• The Goal: To determine if lower-limb effort-versus-rest decoders trained on public healthy EEG data can survive transport to a stroke target domain, while auditing source construction, minimal adaptation burden, and confound sensitivity.

2. Methodology

The study is designed as a retrospective public external benchmark rather than a model-optimization leaderboard.

Datasets

Three public datasets were harmonized to a common binary control variable (Effort vs. Rest):

1. EEGMMIDB (Source): Healthy motor execution/imagery (109 volunteers). Used as a balanced source subset.
2. MILimbEEG (Source): Healthy lower-limb execution/imagery (60 participants). Noted for extreme class imbalance (mostly effort).
3. Stroke2025 (Target): Stroke motor-imagery (27 participants). Served as the sole target domain for transport.

Experimental Design

• Preprocessing: Data was resampled to 125 Hz, band-pass filtered (1–35 Hz), average-referenced, and z-scored per trial. Fixed 2.0s windows were used.
• Source Construction: Six source conditions were tested to evaluate how data composition affects transport:
  • EEGMMIDB-only, MILimbEEG-only, Pooled Raw, Pooled Class-Balanced, Pooled Dataset-Balanced, and Pooled Class+Dataset-Balanced.
• Adaptation Ladder:
  1. Zero-shot: No target data used.
  2. 10-shot Calibration: Temperature scaling using 10 labeled trials per class (target) to adjust operating thresholds without updating model weights.
  3. 10-shot Fine-tuning: Updating model weights using 10 labeled trials per class.
• Models:
  • Classical: CSP + LDA, Riemannian Tangent-Space Logistic Regression (Riemann+TSLR).
  • Deep: EEGNet (primary), plus exploratory Domain Generalization (DG) variants (CORAL, VREx, GroupDRO, DANN).
• Controls:
  • Montage Controls: Compared canonical 16-channel vs. frontal-only, temporal-only, and motor-Laplacian montages to test for motor-specificity.
  • Robustness: Support-set resampling (20 draws), bootstrap confidence intervals, label permutation, and leakage checks.

3. Key Results

A. Zero-Shot Transport is Weak

• Performance: Healthy-to-stroke zero-shot transport was generally poor across all models.
• Best Result: The classical CSP+LDA model achieved the highest AUROC of 0.603.
• Deep Learning Failure: EEGNet performed near chance (AUROC 0.527), significantly underperforming classical baselines.
• Operating Behavior: CSP+LDA was heavily sensitivity-weighted (Sensitivity 0.922, Specificity 0.180), while EEGNet was almost entirely negative-predicting (Sensitivity 0.018, Specificity 0.983).

B. Adaptation Improves Behavior, Not Discrimination

• Calibration: 10-shot temperature calibration drastically improved operating behavior (reducing Expected Calibration Error from 0.267 to 0.035 and increasing specificity to 0.485) but left AUROC essentially unchanged (0.603 $\to$ 0.604).
• Fine-tuning: 10-shot fine-tuning yielded only modest gains in discrimination (AUROC increased from 0.603 to 0.605 for CSP+LDA).
• Implication: Limited subject-specific data helps the model behave more appropriately (better calibration/thresholding) but does not significantly improve its ability to rank signals correctly.

C. Source Construction Matters More Than Model Novelty

• Source Sensitivity: Using MILimbEEG-only as a source resulted in failure (AUROC ~0.46), likely due to extreme class imbalance and task mismatch.
• Balancing: Aggressive class-plus-dataset balancing reduced the source size too much, hurting performance. Pooled Raw and Pooled Dataset-Balanced sources performed best, suggesting that source similarity and composition are more critical than simple balancing.
• Deep Learning: Exploratory domain-generalization variants (DANN, etc.) did not rescue transport; classical models remained superior.

D. Confound Sensitivity (Physiology Audit)

• In within-dataset benchmarks, frontal and temporal montage controls performed competitively (AUROC ~0.71–0.73) compared to the canonical motor montage (AUROC ~0.78).
• Conclusion: The signal is not strictly motor-cortical; it contains significant non-motor confounds (e.g., eye movement, temporal artifacts), limiting claims of brain-specificity.

4. Key Contributions

1. First Public External Benchmark: Established a locked, reproducible framework for testing healthy-to-stroke lower-limb EEG transport using three distinct public datasets.
2. Audit of Source Construction: Demonstrated that how the healthy source data is constructed (composition, balancing) is a first-order variable that determines transport success, often outweighing model architecture choices.
3. Decoupling Calibration from Discrimination: Showed that minimal adaptation (10-shot) primarily fixes calibration and operating points rather than improving the fundamental ranking capability (AUROC) of the decoder.
4. Confound Awareness: Provided evidence that retrospective public data lacks sufficient confound controls (EOG/EMG) to support strong claims of motor-specificity, urging caution in interpreting "brain-specific" signals.

5. Significance and Conclusion

• Clinical Relevance: The study challenges the assumption that healthy-to-stroke transfer is a solved problem. Current performance (AUROC ~0.60) is insufficient for clinical deployment or non-invasive brain-spine interfaces.
• Future Direction: The authors argue that further retrospective model iteration (trying new deep learning architectures) is unlikely to yield decisive gains.
• Recommendation: The field must shift toward harmonized prospective validation. Future studies should:
  • Use locked task ontologies across healthy and stroke cohorts.
  • Include explicit confound channels (EOG, EMG, kinematics).
  • Focus on clinically anchored operating points rather than just AUROC.

Final Verdict: The paper concludes that while lower-limb EEG holds promise, the current "healthy-to-stroke" transport gap is a bottleneck caused by source mismatch and unresolved confounds, not a lack of model complexity. The path forward lies in rigorous prospective study design rather than algorithmic tweaking.