Applicability of memorization indicators for early spotting of overfitting while recalibrating sEMG-decoders on low sample sizes

Technical Summary: Applicability of Memorization Indicators for Early Spotting of Overfitting in sEMG Decoders

Problem Statement
Deep learning models for surface electromyography (sEMG) require subject-specific recalibration to account for individual variations in factors such as age, gender, and electrode placement. However, practical constraints limit the number of movement repetitions a user can perform during calibration, creating a low-sample regime. In this setting, standard overfitting mitigation techniques, such as early stopping based on a validation set, are often infeasible because acquiring additional held-out data prolongs the calibration process or is unavailable. Consequently, there is a risk that fine-tuning a pre-trained decoder on limited data leads to overfitting, potentially degrading performance compared to the uncalibrated model. The paper addresses the need for indicators that can detect unsuccessful learning or overfitting using only the existing training data, without requiring extra validation samples.

Methodology
The authors conducted a transfer-learning experiment using the NinaPro Database 2 (49 movement classes, 12 channels). The experimental design involved two phases:

1. Pre-training: A compact 1D Convolutional Neural Network (CNN) was pre-trained on data from 20 subjects. The architecture consisted of three 1D-CNN blocks followed by three fully connected ReLU layers, designed to be lightweight enough for edge hardware inference.
2. Fine-tuning (Calibration): The pre-trained model was fine-tuned on data from 10 new subjects (21–30). To simulate low-sample conditions, the model was trained on 1 to 4 repetitions per subject. Two fine-tuning strategies were compared:
   • Partial Retraining: Only the last two fully connected layers were unfrozen.
   • Full Retraining: All model parameters were retrained (assumed to be more prone to overfitting).

Memorization Indicators
Instead of monitoring validation loss, the study tracked activation statistics of the Rectified Linear Units (ReLU) in the last hidden layer (256 neurons) using only the training data. The primary metric was the Mean Activation Rate (MAR), calculated as the proportion of samples where a neuron's output was greater than zero. Additional distributional metrics were introduced, including the 25th, 50th (median), and 75th percentiles of activation rates, as well as the coefficient of variation. These metrics were computed on the first batch (64 samples) of each epoch during fine-tuning.

Key Results

• Baseline Performance: The pre-trained model achieved a test accuracy of 0.34 on the pre-training subjects but dropped to 0.065 on unseen subjects, confirming the necessity of recalibration.
• Differentiation of Learning Outcomes: The study observed a clear divergence in activation behavior between successful and unsuccessful fine-tuning:
  • Successful Calibration: Models that improved test accuracy (typically those with only the last layers unfrozen) exhibited a slight positive or stable change in MAR during the initial epochs.
  • Unsuccessful Calibration (Overfitting): Models that failed to generalize or degraded in performance (typically those with all layers retrained) showed a significant and consistent decrease in MAR.
• Correlation: A strong qualitative relationship was found where a drop in test accuracy during fine-tuning was accompanied by a reduction in the activation rates of the last hidden layer. This trend was observable as early as 5 epochs.
• Distributional Metrics: While MAR provided a clear separation, the coefficient of variation and quantile distributions offered additional insights. The coefficient of variation increased during overfitting scenarios, whereas it initially dropped in successful models before rising as the generalization gap widened.

Significance and Claims
The paper claims to provide the first empirical evidence that activation-based memorization indicators can serve as a reliable tool for early spotting of overfitting in low-sample sEMG calibration settings. The primary significance lies in the fact that these indicators:

1. Require No Validation Data: They are computed solely from training data, making them ideal for scenarios where acquiring extra repetitions is impractical.
2. Offer Early Detection: Changes in activation rates precede or accompany performance degradation, allowing for potential early stopping or intervention.
3. Are Lightweight: The metrics are computationally inexpensive, suitable for real-time monitoring on edge devices.

The authors maintain a modest tone regarding their findings, noting that the absolute scale of MAR is dependent on dataset properties (e.g., number of classes, recording setup) and architecture. Therefore, they do not propose universal thresholds but rather demonstrate the feasibility of using these statistics as relative, within-experiment indicators. The work suggests that while the current results are promising for compact architectures, future research is needed to validate these indicators across larger, deeper networks and diverse datasets to establish standardized reference ranges.