Hierarchic-EEG2Text: Assessing EEG-To-Text Decoding across Hierarchical Abstraction Levels

1. Problem Statement

The paper addresses the significant challenge of EEG-to-Text decoding, specifically the difficulty of mapping Electroencephalogram (EEG) signals to fine-grained textual stimuli (words) from a large vocabulary.

• Signal Limitations: EEG has a low signal-to-noise ratio (SNR) and captures spatially averaged neural activity, making it difficult to distinguish between fine-grained concepts (e.g., distinguishing "cat" from "dog" or specific food items).
• The Hierarchy Gap: While cognitive neuroscience suggests the brain processes objects hierarchically (from abstract concepts like "living thing" to specific instances), most existing EEG decoding studies use "flat" label spaces (treating all classes as equally distinct).
• Performance Degradation: As the number of classes increases (e.g., from 10 to 1000+), decoding performance drops to near-chance levels. The authors hypothesize that while EEG may fail at fine-grained classification, it might successfully encode abstract-level representations.

2. Methodology

A. Dataset and Preprocessing

• Dataset: The study utilizes the PEERS dataset (Penn Electrophysiology of Encoding and Retrieval Study), containing ~931,538 EEG samples from 264 participants viewing 1,610 English words representing real-world objects.
• Preprocessing:
  • Channel Alignment: To handle different electrode layouts (129-channel GSN, 128-channel BioSemi), the authors aligned 96 common channels using a $k$-nearest neighbor approach based on 3D electrode coordinates.
  • Filtering: Data was re-referenced to average, downsampled to 200Hz, band-pass filtered (0.3–75Hz), and normalized.
  • Windowing: 1-second EEG windows were extracted for each word.

B. Hierarchical Framework Construction

The core innovation is the construction of a Semantic Hierarchy using WordNet (via NLTK):

• Directed Acyclic Graph (DAG): Words in the dataset are treated as leaf nodes. Synsets and hypernym relationships (IS-A) form the internal nodes, creating a DAG where the root represents the most abstract concept and leaves represent specific words.
• Pruning: Overly broad concepts (e.g., "creature") near the root were removed to ensure meaningful classification tasks.
• Splitting: The data was split into Meta-Train (1,126 words), Meta-Validation (92 words), and Meta-Test (392 words) based on subject and word occurrences to ensure disjoint sets for rigorous evaluation.

C. Episodic Evaluation Strategy

Instead of training on all classes simultaneously, the authors adopted an episodic learning framework:

• Episode Sampling: An "episode" is a small classification task (e.g., $N$-way, $k$-shot) derived from a specific node in the DAG.
  • Span: The set of leaf nodes reachable from an internal node defines the "span." A large span implies a broad, abstract concept (coarse-grained); a small span implies specific, fine-grained concepts.
  • Sampling: Episodes are sampled by selecting internal nodes at various depths. This allows testing the model's ability to classify EEG signals at different levels of abstraction (e.g., classifying "animal" vs. classifying "dog").
• Algorithms Tested:
  • Baseline: Standard non-episodic training evaluated in an episodic manner.
  • Meta-Learning: Model-Agnostic Meta-Learning (MAML) and Prototypical MAML (Proto-MAML), including first-order approximations (fo-MAML).
• Architectures:
  • EEGNet: Traditional convolutional architecture.
  • NICE-EEG: Modern architecture with spatial transformers.
  • CBraMod: A foundational model pre-trained on a large EEG corpus (tested with and without pre-training weights).

D. Metrics

The authors use Normalized Accuracy (Chance-Adjusted Accuracy) to allow fair comparison across $N$-way tasks with varying $N$:
$$\text{Normalized Accuracy} = \frac{A - \frac{1}{N}}{1 - \frac{1}{N}} \times 100$$
Where $A$ is the accuracy. A score of 0 represents random guessing; negative scores indicate performance worse than chance.

3. Key Contributions

1. Hierarchical Episodic Framework: Proposed the largest episodic EEG-to-text framework (931k+ samples) that explicitly samples classification tasks based on semantic hierarchy (WordNet) rather than random class selection.
2. Abstraction Depth Analysis: Systematically demonstrated that EEG decoding performance is highly dependent on the level of semantic abstraction.
3. Empirical Findings on Pretraining vs. Meta-Learning: Showed that in high-noise, real-world EEG tasks, self-supervised pre-trained models (CBraMod) outperform complex meta-learning algorithms (MAML), suggesting that representation quality is more critical than adaptation mechanisms in this domain.
4. Real-World Task Validation: Moved beyond passive viewing of static images to a dataset involving decision-making and arithmetic tasks, reflecting more complex, real-world cognitive states.

4. Results

• Non-Episodic Failure: When evaluated on the full 1,126-class space (non-episodic), all models (including CBraMod) failed to generalize, achieving near-zero accuracy. This confirms the difficulty of fine-grained decoding in large label spaces.
• Episodic Improvement: Episodic evaluation yielded modest but consistent improvements over non-episodic baselines, particularly for CBraMod with pre-trained weights.
• Hierarchy and Abstraction:
  • Coarse-Grained vs. Fine-Grained: Performance was significantly better for coarse-grained tasks (large span sizes, e.g., 100+ classes like "living things") compared to fine-grained tasks.
  • Binary vs. Multi-Class: In 2-way classification, coarse-grained tasks performed best. However, as the number of classes increased (6-way, 10-way), performance dropped for all abstraction levels, likely due to the presence of semantically similar classes within the span.
• Pruning Analysis (Abstraction Levels):
  • When the DAG was pruned (moving up the hierarchy to $L-2, L-3, L-4$), treating internal nodes as classes, performance generally improved.
  • Specifically, models trained on highly abstract classes ($L-4$) outperformed those on specific classes, even when the models were trained from scratch (without pre-training). This confirms that EEG signals encode abstract object representations more robustly than specific word identities.
• Meta-Learning vs. Pretraining: Contrary to expectations in few-shot learning, MAML and Proto-MAML did not outperform the Baseline. The Baseline using pre-trained CBraMod achieved the highest performance. The authors suggest that the high noise in EEG signals makes the "adaptation" step of meta-learning amplify noise rather than extract signal, whereas pre-trained representations provide a robust starting point.

5. Significance and Conclusion

• Theoretical Insight: The paper provides strong evidence that while EEG cannot currently decode specific words from a large vocabulary, it does capture hierarchical, abstract semantic representations. The brain's neural dynamics for object perception are accessible at higher levels of the semantic hierarchy.
• Practical Implication: For Brain-Computer Interface (BCI) development, focusing on abstract concept decoding (e.g., "Is the user thinking of an animal or a vehicle?") is more feasible than fine-grained word decoding.
• Methodological Shift: The study advocates for episodic, hierarchy-aware evaluation as a standard for EEG decoding, moving away from flat, large-scale classification benchmarks that yield near-chance results.
• Future Directions: The authors suggest that future work should explore multi-modal approaches (aligning EEG with text embeddings) and investigate why meta-learning struggles in high-noise EEG environments compared to image-based tasks.

In summary, Hierarchic-EEG2Text reframes the EEG decoding problem from a "can we decode any word?" challenge to a "at what level of abstraction can we decode?" inquiry, demonstrating that abstraction depth is a critical, underexplored dimension for successful neural decoding.