Improving Emotion Classification by Combining fNIRS-Derived Hemodynamic Responses with Peripheral Physiological Signals

This study demonstrates that combining functional near-infrared spectroscopy (fNIRS) with electrodermal activity (EDA) significantly improves subject-independent classification of arousal and valence, achieving the highest performance among tested multimodal configurations.

The Gist

The Big Idea: Reading Minds Without the Mind Reading

Imagine you want to know how someone is feeling. Usually, you ask them, "Are you happy or sad?" or "Are you excited or bored?" But asking people constantly is annoying, and sometimes people aren't honest or don't even realize how they feel.

Scientists want to build "emotion-detecting robots" that can guess your feelings just by looking at your body. To do this, they need to look at two different parts of the human "machine":

1. The Brain (The Boss): What is the central command center thinking?
2. The Body (The Crew): How are the hands, heart, and skin reacting?

This study asked a simple question: If we look at both the Brain and the Body together, can we guess emotions better than if we just look at the Brain?

The Tools: The "Headband" and the "Wristband"

The researchers used two main tools to gather data:

• The fNIRS Headband (The Brain Scanner): This is a wearable hat that shines safe, near-infrared light through your forehead. It's like a flashlight looking for traffic jams. When a part of your brain gets active, blood rushes there (just like cars rushing to a construction site). The headband detects this blood flow. It's great because it doesn't get messed up if you move your head or blink, unlike older brain scanners (EEG) which are very sensitive to noise.
• The Wrist Sensors (The Body Reporters):
  • EDA (Skin Sensor): Measures how sweaty your palms are. Think of this as the body's "excitement meter." When you are nervous or excited, your skin gets a little sticky.
  • PPG (Heart Sensor): Measures your pulse. Think of this as the "rhythm tracker." It sees how fast or slow your heart is beating.

The Experiment: A Music Video Party

The researchers gathered 30 university students and showed them 12 short music videos.

• Some videos were designed to make you feel High Energy (like a rock concert) or Low Energy (like a lullaby).
• Some were designed to make you feel Happy/Positive or Sad/Negative.

While the students watched, the "Headband" and "Wristband" recorded everything. Afterward, the students rated how they felt. The goal was to see if a computer could look at the sensor data alone and correctly guess: "This person is feeling High Energy" or "This person is feeling Sad."

The Results: The Power of Teamwork

The researchers tried different combinations, like a sports coach trying different lineups:

1. Brain Only: The headband tried to guess the mood. It was okay, but not perfect.
2. Body Only: The wrist sensors tried to guess. The skin sensor was good at guessing "High Energy," but bad at guessing "Sad vs. Happy."
3. The Dream Team (Brain + Skin): When they combined the Headband (Brain) and the Skin Sensor (EDA), the computer got the best results.

The Analogy:
Imagine trying to guess what a movie is about.

• Brain Only: You are watching the movie with the sound turned off. You can see the actors' faces, but you miss the dialogue. You can guess the plot, but it's hard.
• Body Only: You are listening to the soundtrack with your eyes closed. You hear the music is loud and fast (High Energy), but you don't know if it's a happy song or a scary one.
• The Dream Team: You have the sound and the picture. You see the actor's face (Brain) and hear the music (Skin). Suddenly, the meaning is crystal clear.

The Key Takeaways

1. Two Heads are Better Than One: Combining brain signals with skin signals (EDA) was the winning strategy. It helped the computer distinguish between "High Energy" and "Low Energy" much better than using just the brain.
2. Different Jobs for Different Sensors:
   • The Skin Sensor was the star for detecting Arousal (how excited or calm you are).
   • The Brain Sensor was crucial for detecting Valence (whether the feeling is good or bad).
   • When you put them together, they covered each other's blind spots.
3. Simplicity Wins: The researchers didn't use complex, "black box" AI. They used simple math and standard tools. This proves you don't need a supercomputer to get good results; you just need the right combination of sensors.

Why Does This Matter?

This research suggests that in the future, we could have wearable devices (like a smartwatch and a comfortable headband) that help computers understand our emotions in real-time.

Because the brain scanner (fNIRS) is tough and doesn't get confused by movement, this system could work in real life—not just in a quiet lab. Imagine a car that knows you are stressed and plays calming music, or a computer that knows you are frustrated and offers help, all without you having to say a word.

In short: To understand human emotion, don't just listen to the brain; listen to the whole body. The brain and the skin work best when they talk to each other.

Technical Summary

1. Problem Statement

The core challenge in affective computing is the development of robust, subject-independent (cross-subject) emotion recognition systems that can operate in naturalistic environments.

• Limitations of Current Modalities:
  • EEG: While sensitive to rapid affective changes, it is highly susceptible to motion artifacts and electromagnetic interference, making it difficult to use outside controlled labs.
  • Peripheral Signals (EDA/PPG): While robust to noise, they often lack the specificity to distinguish complex emotional states (particularly valence) without central nervous system data.
  • fNIRS: Offers a middle ground with better motion tolerance than EEG and direct measurement of brain activity, but its utility in multimodal combinations with peripheral signals for subject-independent classification remains under-explored.
• Research Gap: It is unclear whether combining fNIRS with peripheral autonomic measures (specifically Electrodermal Activity [EDA] and Photoplethysmography [PPG]) yields measurable improvements over unimodal fNIRS for classifying the fundamental affective dimensions of arousal and valence in a subject-independent setting.

2. Methodology

A. Experimental Design & Stimuli

• Participants: 35 healthy Japanese university students (24 male, 11 female; mean age 20.7). After exclusions due to recording failures, 30 subjects were used for analysis.
• Stimuli: 12 Japanese music-video segments (60 seconds each) were selected from an initial pool of 40. Selection was based on high inter-subject consistency in arousal and valence ratings using the Self-Assessment Manikin (SAM) scale.
  • The set included 3 videos for each of four categories: High/Low Arousal and High/Low Valence.
• Protocol: Subjects viewed videos while wearing sensors. Each session included a 20s baseline, video viewing, a 20s rest, and subjective rating.

B. Data Acquisition

Three physiological modalities were recorded simultaneously:

1. fNIRS (Central): A 34-channel wearable system (Hitachi WOT-HS) measuring oxy- and deoxy-hemoglobin concentrations. Channels were mapped to five anatomical regions: Medial Prefrontal, Bilateral Lateral Prefrontal, and Bilateral Temporal areas.
2. EDA (Peripheral): Measured via Ag/AgCl electrodes on the left index and middle fingers (Shimmer3 GSR+).
3. PPG (Peripheral): Measured via an optical pulse sensor on the left index finger.

C. Preprocessing & Feature Extraction

The study prioritized reproducibility and transparency by using simple, standard features rather than deep learning or complex feature engineering.

• fNIRS: Motion artifacts were corrected using a wavelet-based method ($iqr=1.5$). Signals were low-pass filtered (0.5 Hz), detrended, and z-scored. Features extracted per region: Mean, Standard Deviation, and Temporal Slope for both O2Hb and HHb (Total: 30 features).
• EDA: Upsampled to 256 Hz. Decomposed into tonic and phasic components. Features included autocorrelation, distributional stats (SD, skewness, kurtosis) for both components, and peak metrics (count, amplitude) for the phasic component (Total: 9 features).
• PPG: Upsampled to 256 Hz. 27 features extracted using NeuroKit2, comprising 19 time-domain and 8 frequency-domain Heart Rate Variability (HRV) metrics.

D. Classification & Evaluation

• Task: Binary classification (High vs. Low) for Arousal and Valence separately.
• Model: Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel.
• Validation Strategy: Grouped 5-fold Cross-Validation where the "group" was the subject identity. This ensures no data from a single subject appears in both training and testing sets (strictly subject-independent).
• Metrics: Macro-averaged F1 score, Precision, Recall, and Accuracy.
• Statistical Analysis:
  • Permutation Tests: To establish significance against chance (300 permutations).
  • Wilcoxon Signed-Rank Test: For pairwise comparisons between feature sets.
  • Permutation Feature Importance: To identify which specific features drove classification performance.

3. Key Results

A. Classification Performance

The combination of fNIRS + EDA yielded the highest performance for both dimensions:

• Arousal:
  • fNIRS + EDA: Macro-averaged F1 = 0.73 (Best).
  • fNIRS alone: 0.65.
  • EDA alone: 0.69.
  • Full multimodal (fNIRS + EDA + PPG): 0.72.
• Valence:
  • fNIRS + EDA: Macro-averaged F1 = 0.64 (Best).
  • fNIRS alone: 0.58.
  • EDA alone: 0.51 (not significant).
  • Full multimodal: 0.63.

Key Finding: Adding EDA to fNIRS significantly improved performance over fNIRS alone for both arousal and valence. Adding PPG provided marginal or no additional benefit over fNIRS + EDA.

B. Feature Importance Analysis

• Arousal: EDA-derived features (particularly skin conductance response metrics) were the dominant contributors when combined with fNIRS. This aligns with the physiological understanding that EDA is a strong proxy for arousal.
• Valence: The most informative features were a mix of fNIRS (central) and EDA (peripheral). Neither modality alone was sufficient for robust valence classification; they provided complementary information.
• PPG: While significant for arousal, PPG features generally ranked lower than EDA features in the multimodal context.

4. Key Contributions

1. Empirical Validation of Multimodal Fusion: Demonstrated that subject-independent emotion classification is significantly improved by fusing fNIRS (central) with EDA (peripheral), specifically achieving a macro-F1 of 0.73 for arousal and 0.64 for valence.
2. Methodological Transparency: The study deliberately avoided "black box" deep learning approaches. By using simple, standard features and classical SVMs, the authors created a highly reproducible benchmark for future research.
3. Insight into Complementary Modalities: The results suggest that while arousal can often be inferred from a single modality (EDA or fNIRS), valence requires the complementary integration of central hemodynamic signals and peripheral autonomic responses.
4. Practical Feasibility: Highlighted the potential of fNIRS-centered systems for real-world applications due to fNIRS's robustness to motion and electromagnetic noise compared to EEG.

5. Significance & Limitations

• Significance: This work supports the development of wearable affective computing systems that are robust enough for naturalistic settings. It establishes that fNIRS is a viable central sensor that, when paired with simple peripheral sensors like EDA, can effectively decode emotional states across different individuals without requiring subject-specific calibration.
• Limitations:
  • Dataset Demographics: The sample was predominantly young male university students (2:1 ratio), limiting generalizability to other age groups or gender-balanced populations.
  • Feature Complexity: The study intentionally used simple features. Future work could explore if deep learning or sensor-level fusion could further improve performance, though at the cost of interpretability.
  • Peripheral Scope: Only EDA and PPG were tested. Other modalities (ECG, EMG, Skin Temperature) were not evaluated for their synergistic potential with fNIRS.

Conclusion: The study concludes that integrating fNIRS with peripheral physiological signals, particularly EDA, is a highly effective strategy for subject-independent emotion classification, offering a promising path toward robust, real-world affective computing.