Stimulus identity rather than emotion drives EEG classification on the FACED dataset

Technical Summary: Stimulus Identity Rather Than Emotion Drives EEG Classification on the FACED Dataset

Problem Statement
Reliable benchmark datasets are foundational for advancing EEG-based emotion recognition. The Finer-grained Affective Computing EEG Dataset (FACED), currently the largest publicly available EEG emotion dataset comprising 123 subjects and nine emotion categories, has become a widely adopted benchmark. However, this paper identifies a critical flaw in the dataset's utility: classification performance on FACED appears to reflect stimulus identity rather than genuine emotion. The study argues that current evaluation protocols on FACED are susceptible to confounds arising from specific design choices, leading to inflated or misleading accuracy metrics that do not generalize to true emotional decoding.

Methodology
The authors employed a comparative analysis using two distinct classification approaches: a linear classifier (LinearSVC) and a deep learning model (CLISA). To isolate the drivers of classification performance, the study conducted three specific experimental manipulations on the FACED data:

1. Feeling vs. Non-Feeling Analysis: The authors compared classification performance on trials where subjects explicitly reported feeling the assigned emotion against trials where they did not.
2. Label Replacement: The study replaced the original stimulus-assigned labels (the emotion the video was intended to evoke) with individual self-reports of the emotion actually felt by the subject, then re-evaluated classification accuracy.
3. Stimulus Reduction: The authors reduced the dataset to a single video per emotion category, effectively discarding two-thirds of the data, to test whether accuracy persisted when stimulus variability was minimized.

Key Results
The experimental results yielded three primary findings that challenge the validity of FACED as an emotion-recognition benchmark:

• Indistinguishable Performance: Classification performance remained comparable regardless of whether subjects reported feeling the target emotion or not, suggesting the model is not learning emotional states but rather stimulus features.
• Performance Degradation with Self-Reports: When stimulus-assigned labels were replaced with individual self-reports, classification accuracy dropped significantly. This indicates that the models trained on the original labels are failing to capture the subjective emotional experience of the participants.
• Accuracy Increase via Stimulus Reduction: Contrary to the expectation that less data would hurt performance, accuracy increased when the dataset was reduced to one video per emotion. This counter-intuitive result confirms that the models are leveraging stimulus identity (specific visual or auditory features unique to a video) rather than the emotional category itself.

Root Cause Analysis
The paper attributes these findings to three specific design choices inherent to FACED:

1. Few Stimuli per Category: The limited number of unique videos per emotion category creates a strong correlation between the stimulus and the label.
2. Stimulus-Assigned Labels: The reliance on the intended emotion of the video rather than the subject's actual reported feeling introduces a label-noise issue.
3. Within-Video Temporal Splits: The use of temporal splits for cross-validation (splitting data from the same video into training and testing sets) allows the model to learn temporal autocorrelation and specific stimulus features, which then generalize perfectly to the test set without learning the underlying emotion.

Significance and Recommendations
The paper concludes that current benchmarks on FACED are likely measuring the ability to recognize specific video clips rather than decoding emotional states. To guide future research and prevent these confounds, the authors propose five specific recommendations for emotion-decoding study designs:

• Enhance stimulus diversity to ensure multiple unique stimuli per emotion category.
• Implement temporal independence in data splitting to prevent leakage of stimulus-specific features.
• Prioritize label validation using self-reports rather than relying solely on stimulus-assigned labels.
• (Implicitly) Design experiments that explicitly test for and mitigate stimulus-identity confounds.

The significance of this work lies in its call to re-evaluate the validity of widely used benchmarks. By exposing the susceptibility of FACED to stimulus-identity confounds, the paper aims to steer the community toward more rigorous experimental designs that genuinely advance the field of EEG-based emotion recognition.