FingerFlex: Inferring Finger Trajectories from ECoG signals

Technical Summary: FingerFlex: Inferring Finger Trajectories from ECoG signals

Problem Statement
The development of motor brain-computer interfaces (BCIs) relies on decoding neural time series to restore lost functions or augment abilities via external device control. While non-invasive methods like EEG and MEG lack spatial resolution, and invasive single-unit recordings are highly invasive, electrocorticography (ECoG) offers a middle ground with high spatial resolution and signal quality. The specific challenge addressed in this work is the regression of continuous finger trajectories based solely on ECoG-registered oscillatory patterns in the sensorimotor cortex. Existing solutions often rely on rigorous manual feature selection or conventional machine learning models, which may not fully capture higher-order dependencies in the data.

Methodology
The authors propose FingerFlex, a convolutional encoder-decoder architecture adapted for finger movement regression. The methodology comprises three main components: data preprocessing, model architecture, and training strategy.

1. Data Preprocessing and Feature Extraction:

   • ECoG Processing: Raw ECoG signals (sampled at 1000 Hz) undergo standardization, median subtraction, and band-pass filtering (40–300 Hz) to isolate gamma and high-gamma components, which correlate significantly with movement. Complex Morlet wavelets (40 kernels, logarithmically spaced from 40–300 Hz) are convolved with the signals to extract time-frequency representations (spectrograms). The data is downsampled to 100 Hz via decimation and standardized using RobustScaler to handle outliers. A forward shift (delay hyperparameter, 0–200 ms) is applied to account for the physiological lag between brain activity and movement.
   • Finger Movement Processing: Finger trajectory data (sampled at 25 Hz) is upsampled to 100 Hz using bicubic interpolation and scaled using MinMax scaling.

2. Model Architecture:

   • The model is a convolutional autoencoder with skip connections, inspired by the UNet and wav2vec architectures.
   • Encoder: Consists of 1D convolutional layers, layer normalization, GeLU activation, dropout, and maximum pooling (stride 2). It reduces dimensionality and encodes time window fragments.
   • Decoder: A symmetrical set of convolutional decoder layers that utilize information from the previous layer and corresponding encoder layers via skip connections. This allows the model to use both generalized latent representations and original local features.
   • Output: A 1×1 convolution layer predicts the coordinates for five fingers at each time point.
   • The model processes arbitrary-length windows, though training used fixed windows of 256 time points (2.56 seconds).

3. Training and Loss Function:

   • Loss Function: A hybrid loss function combining Mean Squared Error (MSE) and Mean Cosine Distance. The authors found that while MSE alone yielded better results than cosine distance alone, the combination improved performance by an average of 0.02 in correlation coefficient.
   • Hyperparameters: The encoder feature dimensions were set to (32, 32, 64, 64, 128, 128). The learning rate was fixed at $8.4 \cdot 10^{-5}$ based on automatic finder recommendations. The delay hyperparameter was set to 20 ms.
   • Training Protocol: Due to shuffled electrode positions in the dataset, models were trained individually for each subject. Hyperparameter tuning was performed on the first subject, and the same model configuration was applied to the other subjects without further adjustment.

Key Contributions

• Architecture: Introduction of a hybrid encoder-decoder architecture with skip connections for ECoG decoding, allowing for the integration of local features with generalized representations.
• Feature Engineering: Explicit extraction of time-frequency features using Morlet wavelets in the gamma/high-gamma range, combined with robust scaling to handle signal variability.
• Loss Function: Demonstration that a combined MSE and cosine distance loss function outperforms either metric individually for this regression task.
• Efficiency: The model is lightweight (~600,000 parameters) and does not require separate decoders for resting and moving states, facilitating potential real-time deployment.

Results
The model was evaluated on the BCI Competition IV dataset 4, which contains ECoG recordings from three patients.

• Performance on BCI Competition IV Dataset 4: FingerFlex achieved a mean correlation coefficient (CC) of 0.67 across all subjects. Individual subject performances were 0.66 (S1), 0.62 (S2), and 0.74 (S3). This outperforms previous state-of-the-art methods, including the competition winner (Liang and Bougrain, 2012; mean CC 0.48) and recent deep learning approaches (e.g., Yao et al., 2022; mean CC 0.53).
• Ablation Studies: The study confirmed that feature scaling, architecture refinement, frequency range adjustment, and skip connections each contributed incrementally to performance improvements. The delay hyperparameter was found to be stable within the 0–200 ms range.
• Cross-Dataset Validation: The model was tested on the Stanford dataset (Miller, 2019) without hyperparameter re-tuning. Results varied by subject (mean CC ranging from 0.22 to 0.71), with lower performance attributed to potential misalignment of ECoG grids relative to motor areas.

Significance and Claims
The authors claim that FingerFlex provides a pathway for developing fully functional, high-precision cortical motor BCIs. The model’s ability to decode continuous trajectories with high correlation, its lightweight nature, and its reliance on current time windows (rather than future data) make it suitable for real-time inference applications. The study emphasizes that the proposed hyperparameters are stable across participants and datasets, reducing the need for extensive per-subject tuning. The work positions itself as a significant improvement over existing conventional machine learning and deep learning solutions for finger trajectory decoding.