MAD: A Multimodal and Multi-perspective Affective Dataset with Hierarchical Annotations

The paper introduces MAD, a novel multimodal affective dataset featuring synchronized physiological signals (EEG, ECG, EOG, EMG, PPG, BCG) and tri-view RGB-D facial videos from 18 participants, accompanied by a hierarchical three-level annotation framework to facilitate comprehensive research in emotion recognition and cross-modal affective analysis.

The Gist

Imagine you are trying to understand why a friend is feeling sad.

In the past, scientists trying to decode human emotions had to rely on two very limited ways of looking at the problem:

1. The "What You See" Method: Watching their face. But people can fake a smile or hide a frown.
2. The "What You Feel" Method: Asking them, "How do you feel?" But people might not know exactly what they feel, or they might lie.

This new paper introduces MAD (Multimodal Affection Dataset), which is like a super-powered, high-tech detective kit for understanding emotions. Instead of just looking at the face or asking a question, MAD records everything happening inside and outside the body at the exact same time.

Here is a simple breakdown of what makes this dataset special, using some everyday analogies:

1. The "Full-Body Orchestra" (The Data)

Imagine an orchestra. Usually, researchers only listen to the violins (the brain/EEG) or just the drums (the heart/ECG). But emotions are a symphony played by the whole band.

MAD records the entire orchestra simultaneously:

• The Conductor (Brain/EEG): What the brain is thinking.
• The Rhythm Section (Heart/ECG, PPG, BCG): How fast the heart is beating and how blood is flowing.
• The Wind Instruments (Muscles/EMG, Eye Movements/EOG): Tiny muscle twitches and eye blinks that happen before you even realize you're reacting.
• The Visuals (3D Cameras): Not just a flat photo, but a 3D video of the face from three different angles (left, front, right), so you can see the emotion even if the person turns their head.

The Magic: Because all these instruments are recorded at the exact same time (synchronized), scientists can see how a thought in the brain instantly triggers a heartbeat change, which then leads to a facial expression. It's like watching the sheet music, the conductor's baton, and the audience's reaction all in one perfect video.

2. The "Three-Layer Cake" (The Labels)

One of the biggest problems in emotion research is that people often feel one thing but show another. MAD solves this by labeling the data in three layers, like a three-tiered cake:

• Layer 1: The Stimulus (The Movie): "This movie clip is supposed to be scary." (The external trigger).
• Layer 2: The Cognition (The Feeling): "I felt terrified while watching that." (The internal, subjective experience).
• Layer 3: The Expression (The Face): "You looked scared in the video." (The outward behavior).

Why this matters: Sometimes the movie is scary (Layer 1), but you feel bored (Layer 2), yet you still frown because you think you should look scared (Layer 3). MAD captures all three layers, allowing scientists to study the gap between what we feel, what we show, and what actually happened.

3. The "Stress Test" (The Experiments)

The authors didn't just collect the data; they put it through a rigorous "stress test" to prove it works. They ran five different experiments, which you can think of as:

• The "Mind-Reading" Test: Can a computer guess your emotion just by looking at your brainwaves? (Yes, and it works better when we know what movie you were watching).
• The "Stranger Danger" Test: Can a model trained on your brainwaves guess the emotions of a stranger? (Yes, MAD helps computers learn to generalize across different people).
• The "Heartbeat" Test: Can we tell emotions just by looking at the heart, without needing brain sensors? (Yes! The paper found that heart sensors like PPG and BCG are surprisingly good at this, which is great for wearables like smartwatches).
• The "Teamwork" Test: What happens if we combine the brain, heart, and muscle signals? (They work better together than alone, like a sports team where the goalie and the striker help each other).
• The "360-Degree" Test: If a person turns their head, does the computer still know they are happy? (Yes, because MAD uses 3D cameras from three angles, teaching the AI to recognize emotions from any side).

The Big Picture: Why Should You Care?

Think of MAD as the "Google Maps" for human emotions. Before this, we were trying to navigate the complex landscape of feelings with a blurry, old paper map.

• For Doctors: It could help detect depression or anxiety earlier by spotting subtle changes in heart rate or brain waves before a person even says they feel bad.
• For Tech: It could lead to cars that know you are stressed and automatically adjust the music or AC, or VR games that get harder when you get bored.
• For Science: It helps us understand the mechanics of emotion. It answers the question: "Does the heart race because we are scared, or do we feel scared because our heart races?"

In short, MAD is a massive, high-quality library of human feelings that records the "inside" (brain/heart) and the "outside" (face) simultaneously. It's a tool that helps us move from guessing how people feel to actually knowing it, scientifically and accurately.

Technical Summary

Here is a detailed technical summary of the paper "MAD: A Multimodal and Multi-perspective Affective Dataset with Hierarchical Annotations."

1. Problem Statement

Current affective computing research faces three primary limitations regarding existing public datasets (e.g., DEAP, SEED, FER2013):

• Limited Modality Coverage: Most datasets focus on a single modality (usually EEG or 2D facial video) or a small subset of peripheral signals. This hinders the study of interactions between central neural activity, autonomic physiological responses, and overt behavioral expressions.
• Single-Level Annotation: Existing datasets typically provide only one type of emotion label (either stimulus-based or self-reported). This prevents researchers from analyzing the consistency or divergence between external stimulus elicitation, internal subjective cognition, and observable behavioral expression.
• Visual Constraints: Visual data is often restricted to single-view 2D RGB videos, limiting research on cross-view robustness and 3D structural perception of facial expressions.

Consequently, there is a lack of a unified, high-precision synchronized dataset that supports cross-level (stimulus-cognition-expression) and cross-modal (neural-physiological-behavioral) investigation of emotional processes.

2. Methodology and Dataset Design

The authors introduce MAD (Multimodal Affection Dataset), a comprehensive resource designed to address these gaps.

A. Data Acquisition

• Participants: 18 healthy adults (9 male, 9 female, aged 23–25).
• Stimuli: 16 emotion-eliciting video clips covering 7 categories (Happy, Angry, Sad, Fearful, Anxious, Disgusted, Neutral).
• Synchronization: A unified timestamp system synchronizes all data streams.
• Modalities Recorded:
  • Physiological (6 types):
    • EEG: 14-channel (Emotiv Epoc+, 128 Hz).
    • ECG/EOG/EMG: BIOPAC MP150 system (2 ECG, 1 EOG, 3 EMG channels, 1000 Hz).
    • Cardiac-related: Custom dual-channel PPG and seat-based BCG sensors (34 Hz).
  • Visual: Tri-view RGB-D videos captured by three Intel RealSense D455 cameras (Left, Frontal, Right) at 640×480 resolution, 30 fps.
• Scale: 288 sessions total, occupying ~8 TB of storage.

B. Hierarchical Annotation Framework

MAD introduces a unique three-level annotation scheme to capture the full emotion process:

1. Stimulus Level (Multimedia): Labels assigned by 50 external annotators based on video content (7-class, 4-class, and 3-class granularities).
2. Cognitive Level (Subjective): Self-reported emotional states and intensity ratings provided by participants after viewing.
3. Expression Level (Behavioral): Labels assigned by 10 observers based on tri-view facial frames (Positive, Neutral, Negative).

C. Benchmark Experiments

The authors established five core benchmark tasks to validate the dataset:

1. Intra-subject EEG Recognition: Comparing performance using stimulus-based vs. cognitive labels.
2. Cross-subject EEG Transfer: Evaluating domain adaptation methods (DAN, DANN, etc.) to handle inter-individual variability.
3. Cardiac Signal Consistency: Analyzing the rhythmic alignment and classification performance of ECG, PPG, and BCG.
4. Multimodal Physiological Fusion: Testing feature-level fusion of EEG, ECG, EOG, and EMG.
5. Multi-view Facial Recognition: Evaluating pose robustness using tri-view data and contrastive learning.

3. Key Contributions

1. Cross-Level Multimodal Dataset: MAD is the first dataset to integrate central neural signals (EEG), diverse peripheral physiological signals (ECG, EOG, EMG, PPG, BCG), and multi-view 3D facial videos with precise temporal synchronization.
2. Hierarchical Annotation Framework: The three-level labeling scheme (Stimulus, Cognition, Expression) enables quantitative analysis of the "elicitation–cognition–expression" gap, allowing researchers to study emotion regulation and subjective variability.
3. Comprehensive Benchmark Protocols: The paper provides standardized benchmarks for intra/cross-subject EEG, cardiac signal analysis, multimodal fusion, and multi-view vision, demonstrating the dataset's utility for diverse research questions.

4. Experimental Results

• Label Consistency: Models trained on stimulus-based labels significantly outperformed those trained on cognitive labels (e.g., 3-class EEG accuracy: 88.8% vs. 69.3%). This suggests EEG responses to external stimuli are more consistent than internal subjective experiences.
• Cross-Subject Transfer: Domain adaptation methods achieved stable performance (approx. 60–65% accuracy) on MAD, indicating that while inter-subject variability exists, it is "realistic yet learnable."
• Cardiac Signals: ECG, PPG, and BCG showed highly consistent peak-to-peak intervals and fluctuation trends. BCG (non-contact) achieved performance comparable to contact-based ECG/PPG, validating its potential for wearable/passive sensing.
• Multimodal Fusion:
  • EEG remained the most discriminative single modality.
  • Peripheral signals (ECG, EOG, EMG) individually performed lower but showed complementarity; fusing them improved robustness, though they did not surpass EEG alone.
• Multi-view Vision:
  • Baseline transfer from FER2013+ was poor (~45% accuracy).
  • Multi-view fine-tuning significantly improved side-view accuracy.
  • Contrastive learning (aligning features across views) achieved the best results (Frontal: 84.5%, Side: 81.8%), proving the effectiveness of learning view-invariant representations.

5. Significance and Impact

• Theoretical Insight: MAD provides empirical evidence for the hierarchical separation between "perception" (stimulus) and "experience" (cognition), supporting psychological theories regarding emotion regulation and the divergence between internal states and external expressions.
• Methodological Advancement: By validating the substitutability of non-contact signals (BCG, PPG) and the efficacy of multi-view contrastive learning, the dataset guides the development of more robust, real-world affective computing systems.
• Resource for Future Research: MAD serves as a unified platform for studying emotion mechanisms across neural, physiological, and behavioral domains. It facilitates research into interpretable AI, cross-modal fusion, and the development of personalized emotion models.
• Benchmark Standard: The dataset establishes a new standard for evaluating emotion recognition algorithms under complex, multi-perspective, and multi-modal conditions, moving beyond controlled single-modality studies.