Copilot-Assisted Second-Thought Framework for Brain-to-Robot Hand Motion Decoding

1. Problem Statement

The primary challenge addressed is the accurate decoding of continuous 3D hand kinematics (specifically thumb and index finger trajectories) from non-invasive Electroencephalography (EEG) signals for Brain-Computer Interface (BCI) applications.

• Limitations of Current Methods: Traditional approaches relying on CNNs or RNNs struggle with the low signal-to-noise ratio (SNR) of EEG, spatial localization issues, and the inability to model long-range temporal dependencies effectively.
• Generalization Gap: Existing models often suffer from poor cross-subject performance due to inter-subject variability in brain signal patterns.
• Trajectory Fidelity: Decoded trajectories often contain noise, reversals, and jitter, making them unsuitable for direct control of robotic arms without significant post-processing.

2. Methodology

The authors propose a three-pronged approach: a hybrid deep learning model, multimodal fusion, and a "Copilot" post-processing framework.

A. Hybrid CNN-Attention Model Architecture

The core decoding model is a hybrid architecture designed to capture both local temporal features and long-range dependencies:

1. Multi-Convolution Block: Inspired by EEGNet, it uses a large temporal kernel to capture long-range dependencies followed by smaller kernels of varying sizes to extract multi-scale temporal features.
2. Squeeze-and-Excitation (SE) Block: A mechanism to recalibrate channel-wise feature responses, highlighting informative channels and suppressing less relevant ones.
3. Embedding & Self-Attention: The channel-feature dimension is abstracted into tokens. A multi-head self-attention mechanism (Transformer-based) establishes long-range dependencies between these tokens, overcoming the limitations of RNNs in handling sequential EEG data.
4. Fully Connected Head: Global average pooling and dense layers map the attention output to 3D coordinates (X, Y, Z) for the thumb and index finger.

B. Multimodal Fusion (EEG + EMG)

To improve robustness, the system integrates Electromyography (EMG) signals alongside EEG.

• Processing: EMG signals undergo similar preprocessing (band-pass filtering, downsampling) and pass through the same Multi-Convolution and SE blocks.
• Fusion: EEG and EMG features are fused within the embedding module before the attention mechanism, allowing the model to leverage muscle activity signals to refine brain signal decoding.

C. Copilot-Assisted Second-Thought Framework

To address trajectory noise and reversals, the authors introduce a post-processing "Copilot" system:

• Finite State Machine (FSM): The grasp-and-lift task is segmented into five states: Searching, Lifting, Holding, Putting, and Returning.
• Critic Model: A secondary model (identical in structure to the decoder) acts as a "critic," estimating confidence scores for each decoded point.
• Knowledge Graph & Filtering: The system uses a knowledge graph to interact with external sensor data and drive state transitions. Points with low confidence or those misclassified as belonging to the current state (labeled as "UNRELY") are filtered out based on state-specific thresholds. This acts as a "second thought" to validate the primary decoding.

3. Key Contributions

1. Novel Architecture: Development of a CNN-Attention hybrid model that outperforms standard EEGNet, DeepConvNet, and pure Transformer baselines for 3D hand kinematics prediction.
2. Multimodal Fusion: Demonstration that fusing EEG with EMG significantly boosts decoding accuracy and generalization, achieving near-perfect within-subject correlation.
3. Copilot Framework: Introduction of a state-aware filtering mechanism that improves trajectory fidelity by removing low-confidence points without discarding excessive data.
4. Robotic Simulation: Successful reconstruction of grasp-and-lift movements on a Franka Panda robotic arm in a MuJoCo simulation, validating the practical applicability of the decoded trajectories.

4. Experimental Results

The study utilized the WAY-EEG-GAL dataset (12 participants, 3,936 trials of grasp-and-lift tasks).

A. Within-Subject Performance

• Multimodal (EEG+EMG): Achieved exceptional performance with Pearson Correlation Coefficients (PCC) of 0.9854 (X), 0.9946 (Y), and 0.9065 (Z).
• EEG-Only: The proposed hybrid model achieved a PCC of 0.8728 (0.8376, 0.9229, 0.7208 for X, Y, Z), outperforming the Transformer baseline (0.8353) and other CNN/RNN models.
• Copilot Impact: Post-filtering improved the overall EEG-only PCC to 0.93 while retaining over 80% of the data points (filtering <20%).

B. Cross-Subject Performance

• Multimodal: Showed strong generalization with a total PCC of 0.9063 (0.9643, 0.9795, 0.5852 for X, Y, Z).
• EEG-Only: Performance dropped significantly (Total PCC ~0.66), highlighting the difficulty of subject-independent decoding, though the multimodal approach mitigated this drop.

C. Ablation Studies

• Window Size: Performance peaked at a 750-sample input window (PCC 0.8916), indicating a need for sufficient temporal context.
• Delay: The optimal temporal delay between EEG and kinematics was found to be 200 ms.

5. Significance and Future Work

• Significance: This work bridges the gap between raw EEG decoding and practical robotic control. It demonstrates that hybrid deep learning models combined with multimodal data can achieve high-fidelity trajectory reconstruction. The "Copilot" framework offers a novel solution to the noise problem in BCIs without requiring retraining of the core decoder.
• Limitations: The system still struggles with low-accuracy trajectories where the copilot can only filter but not correct points. Cross-subject performance for EEG-only remains a challenge.
• Future Directions: The authors plan to integrate additional sensors for motion-pattern-specific assistance and explore more robust methods for controlling robotic arms in real-world scenarios.

Code Availability: The source code is publicly available at https://github.com/ssshiFang/EEGDecoding Robotarm.