egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks

This paper introduces egoEMOTION, the first dataset combining egocentric visual and physiological signals with self-reported emotion and personality data from 43 participants, to establish new benchmarks for affect and trait recognition in real-world scenarios.

The Gist

Imagine you are wearing a pair of high-tech glasses that don't just record what you see, but also "feel" what you feel. That is the core idea behind egoEMOTION, a new project by researchers at ETH Zurich.

Here is the story of the paper, broken down into simple concepts and everyday analogies.

1. The Problem: The "Robot" View of the World

For years, computers have gotten really good at watching people through cameras. They can tell if you are walking, cooking, or playing tennis. But there's a big blind spot: The computer doesn't know how you feel while doing those things.

Imagine a robot watching you drive a car. It sees you turning the wheel and pressing the gas. But it doesn't know if you are driving calmly because you are relaxed, or frantically because you are terrified of being late. Current technology assumes everyone is a "neutral robot" with no mood swings or personality quirks. The researchers say, "We need to fix this. To truly understand human behavior, we need to understand the emotional engine driving the car."

2. The Solution: The "Emotion Detective" Kit

To solve this, the team created egoEMOTION, which is essentially a massive library of data. Think of it as a "training manual" for computers to learn about human feelings.

They gathered 43 volunteers and gave them a special toolkit:

• The Glasses (Project Aria): These look like normal sunglasses but have cameras inside. They record what the person sees (Point-of-View), where their eyes are looking, and even track their head movements.
• The Body Sensors: The volunteers also wore a chest strap (like a heart-rate monitor), a belt for breathing, and sensors on their ears and fingers to measure sweat and blood flow.
• The "Check-In" App: After every activity, the volunteers had to stop and tell the computer exactly how they felt using a digital mood wheel.

3. The Experiment: A Day in the Life of an Emotion

The volunteers didn't just sit in a lab staring at a wall. They did two types of activities:

• Session A (The Movie Night): They watched 9 short video clips designed to trigger specific feelings. One clip was a funny scene from When Harry Met Sally (to make them laugh), another was a scary scene from Psycho (to make them scream), and another was a sad scene from Jojo Rabbit.
• Session B (The Real World): They did 7 everyday tasks. They played a game of Flappy Bird, tried to build a tower of blocks (Jenga), painted a picture, and even tried to make each other laugh with jokes.

The Analogy: Think of Session A as a "taste test" where you try distinct flavors (sweet, sour, spicy). Session B is like a full dinner party where you are eating, talking, and laughing all at once. The researchers wanted to see if the computer could detect feelings in both the controlled "taste test" and the messy "dinner party."

4. The Big Discovery: Eyes Are Better Than Heartbeats

The most surprising finding of the paper is about how the computer guessed the emotions.

Usually, scientists think that to know how someone feels, you need to measure their heart rate or sweat (physiological signals). It's like trying to guess if someone is nervous by checking if their hands are shaking.

But egoEMOTION found something different:
The computer was actually better at guessing emotions just by looking at the video from the glasses (specifically where the eyes were looking and how the pupils changed) than by looking at the heart rate sensors.

• The Metaphor: Imagine trying to guess if a friend is bored at a party.
  • Physiological approach: You check their pulse to see if it's racing.
  • Egocentric approach: You look at their eyes. Are they scanning the room? Are they staring at the floor? Are their pupils wide?
  • The Result: The researchers found that looking at the eyes (the "Egocentric" view) gave a clearer picture of the mood than checking the pulse.

5. Why This Matters

This dataset is a game-changer for three reasons:

1. It's the First of Its Kind: It's the first time we have a huge dataset that combines "what you see" (video) with "what you feel" (emotions) and "who you are" (personality) all at the same time.
2. It's Real-World: Unlike old studies where people sat still in a lab, this data comes from people moving around, playing games, and reacting naturally.
3. It Changes How AI Works: It suggests that future AI assistants (like smart glasses or robots) won't need to strap sensors to your chest to know you're stressed. They can just "watch" your eyes and body language to understand you.

Summary

egoEMOTION is like giving a computer a pair of "emotional eyes." It teaches machines that human behavior isn't just about what we do (walking, talking, working), but how we feel while doing it. And the best part? The computer can learn to read our feelings just by watching our eyes, without needing to hook us up to a bunch of wires.

This opens the door to a future where your technology understands you better, helping to create smarter, more empathetic robots and apps that can actually "get" you.

Technical Summary

1. Problem Statement

Current benchmarks in egocentric vision (first-person perspective) primarily focus on external behaviors such as action recognition, object manipulation, and gaze prediction. These systems implicitly assume that human participants have neutral affect and uniform personality traits. This assumption limits the ability of vision systems to model the internal drivers of human behavior (mood, arousal, personality) which are critical for understanding decisions, social interactions, and intent. Furthermore, existing emotion recognition datasets often rely on controlled laboratory settings with limited ecological validity or lack the integration of egocentric visual data with physiological signals in real-world scenarios.

2. Methodology

Dataset Construction (egoEMOTION)

The authors present egoEMOTION, the first multimodal dataset coupling egocentric vision with physiological signals for emotion and personality recognition.

• Participants: 43 healthy adults (ages 19–29).
• Duration: Over 50 hours of synchronized recordings.
• Hardware:
  • Egocentric Vision: Meta's Project Aria glasses capturing:
    • Eye-tracking video (640×480 @ 90 fps per eye).
    • Point-of-View (POV) RGB camera (1408×1408 @ 10 fps).
    • Head IMU (1000 Hz and 800 Hz).
    • Integrated nosepad PPG sensor (128 Hz).
  • Physiological Sensors:
    • ECG: Chest strap (Movisens ECG4Move4, 1024 Hz).
    • EDA: Fingertip sensor (Shimmer3, 256 Hz).
    • Respiration: Respiratory belt (plux respiBAN, 400 Hz).
    • Ear PPG: Shimmer3 unit (256 Hz).
    • External: Webcam (60 fps) for facial expression labeling.
• Experimental Protocol: Two sessions per participant:
  • Session A (Induced): 9 video clips designed to elicit specific emotions (e.g., fear, amusement, disgust) based on Mikels' Wheel, interspersed with washout periods.
  • Session B (Naturalistic): 7 real-world activities (e.g., painting, playing Jenga, solving puzzles, watching a movie scene) to capture spontaneous emotional responses.
• Ground Truth Annotation:
  • Continuous Affect: Self-reported using the Circumplex Model (Valence, Arousal, Dominance) via an emoji-based emoti-SAM.
  • Discrete Emotion: Weighted distribution across 9 categories (Amusement, Content, Excitement, Awe, Fear, Sadness, Disgust, Anger, Neutral) using a weighted Mikels' Wheel.
  • Personality: Assessed via the Big Five Inventory-2 (BFI-2) (Extraversion, Agreeableness, Conscientiousness, Negative Emotionality, Open-Mindedness).

Feature Extraction & Baselines

The authors extracted 612 features across modalities:

• Physiological: Statistical descriptors (mean, std, spectral power, HRV) for ECG, EDA, RSP, and PPG.
• Egocentric Vision:
  • Eye-tracking: Pupil size, gaze (yaw/pitch), blink rate.
  • Visual: Mean pixel intensity, Fisherface features (PCA+LDA), and micro-expressions (LBP-TOP).
  • Motion: IMU acceleration magnitude.
• Models:
  • Classical ML: Support Vector Machines (SVM) for continuous affect; Random Forests for discrete emotion and personality.
  • Deep Learning: Comparison against a CNN and a Transformer-based architecture (WER) to test state-of-the-art performance.
• Validation: Leave-One-Subject-Out (LOSO) cross-validation to ensure generalization across individuals.

3. Key Contributions

1. First Multimodal Egocentric Dataset: Introduces the first public dataset combining egocentric vision (Project Aria) with high-fidelity physiological signals for emotion and personality recognition in both induced and naturalistic settings.
2. Rich Annotation Scheme: Moves beyond binary labels by providing dense self-reports using the Circumplex Model, weighted emotional tags (capturing mixed emotions), and Big Five personality traits.
3. Benchmark Tasks: Defines three specific prediction tasks:
   • Continuous affect regression (Valence, Arousal, Dominance).
   • Discrete emotion classification (9 classes).
   • Trait-level personality inference.
4. Open Source: The dataset and baseline implementations are released under ethical approval (ETH Zurich) with strict access controls to protect privacy.

4. Key Results

The paper presents surprising findings regarding the efficacy of egocentric signals versus traditional physiological sensors:

• Egocentric Signals Outperform Physiological Baselines:
  • Continuous Affect: Using only egocentric signals (glasses), the model achieved an F1 score of 0.74, slightly outperforming traditional wearable physiological signals (0.70) and matching the fused multimodal result (0.75). Pupil size was a particularly strong feature.
  • Discrete Emotion: Egocentric signals alone achieved an F1 of 0.46, significantly higher than the random baseline (0.11) and wearable-only signals (0.24). IMU data alone reached 0.44.
  • Personality: Egocentric signals achieved an F1 of 0.57, outperforming wearables (0.50). Eye gaze was the best individual modality.
• Classical vs. Deep Learning:
  • Surprisingly, classical machine learning methods (SVM/Random Forest) outperformed deep learning models (CNN and Transformer) across all tasks.
  • Example: For continuous affect, Classical ML reached 0.75 F1, while the best Deep Learning model reached 0.68.
  • Reasoning: The authors suggest that the limited dataset size (43 subjects) may hinder deep learning models, which require massive data to generalize, whereas classical methods with hand-crafted features are more robust in this regime.
• Feature Importance: Eye-tracking features (pupil size, gaze) and head motion (IMU) were found to be highly informative, often surpassing ECG and EDA in predictive power for real-world tasks.

5. Significance and Impact

• Paradigm Shift: The results challenge the assumption that physiological sensors are the gold standard for emotion recognition in mobile settings. They demonstrate that egocentric vision systems (like smart glasses) can capture rich internal states (affect and personality) without the need for intrusive chest straps or chest-worn sensors.
• Real-World Applicability: By validating performance in naturalistic tasks (Session B), the dataset proves that affective modeling is feasible outside the lab, enabling "in-the-wild" applications.
• Future Directions: The work opens new avenues for affect-aware Human-Computer Interaction (HCI), adaptive AI, and mental health monitoring. It suggests that future systems can infer user intent and emotional state purely from the visual and motion data captured by wearable glasses, making such technology more practical and less obtrusive.
• Ethical Considerations: The authors highlight the dual-use risk (e.g., behavioral profiling) and enforce strict access controls (academic staff only) to ensure responsible use of the sensitive data.

In conclusion, egoEMOTION establishes that egocentric vision is a powerful, non-invasive modality for understanding human affect and personality, potentially rendering traditional physiological sensors secondary for many real-world applications.