Technical Summary: InCarEmo

Problem Statement

Current research in in-cabin affective computing and driver state monitoring is hindered by a lack of suitable public datasets. Existing resources are often limited in scale, lack modality diversity (relying primarily on visual data), or fail to capture conversational contexts. Specifically, while broader multimodal emotion datasets exist (e.g., IEMOCAP, MELD), they are not grounded in driving-specific scenarios such as traffic conditions, navigation, or vehicle status. Consequently, these datasets cannot effectively capture the linguistic and interactive cues that underlie driver emotions in real-world intelligent cockpit environments. Furthermore, many existing datasets focus on Western participants, limiting the generalizability of models across different cultures and driving contexts.

Methodology

Dataset Construction (InCarEmo)

The authors introduce InCarEmo, a multimodal dataset designed for in-cabin emotion recognition and driver state monitoring. The data collection process involved:

• Setting: Realistic in-cabin environments with vehicles parked to ensure safety while maintaining authentic lighting and acoustic conditions.
• Modalities: The dataset integrates RGB video, Infrared (IR) video, high-fidelity in-cabin audio, and dialogue text.
• Scenarios: Scripts were generated using GPT-4o based on manually designed topics (e.g., traffic, trip planning, vehicle status) to simulate natural driver-passenger or driver-assistant interactions. Approximately 2,000 multi-turn dialogue scripts were created.
• Participants: 25 participants were recruited to ensure diversity in facial characteristics and speaking styles.
• Annotation: Expert annotators reviewed all clips for quality and consistency. Final labels were assigned based on the consensus of three independent experts, achieving a Cohen's kappa coefficient of 0.87.
• Scope: The dataset covers three primary tasks:
  1. Multimodal Emotion Recognition: Classifying six emotions (neutral, happy, sad, angry, surprised, anxious).
  2. Fatigue Detection: Identifying behaviors like eye closure and yawning.
  3. Distraction Monitoring: Detecting behaviors such as drinking, phone usage, and smoking.
• Cross-Lingual Extension: An auxiliary English benchmark was constructed by translating Chinese annotations and synthesizing English speech using the higgs-audio framework, followed by rigorous filtering to ensure quality.

Proposed Baseline: CAMEL

To establish a benchmark, the authors propose CAMEL (Contrastive Alignment and Multi-classifier Ensemble Learning), a lightweight multimodal system tailored for resource-constrained in-vehicle environments.

• Architecture:
  • Backbones: Uses CLIP (for RGB/IR), Chinese-HuBERT (for audio), and Qwen3 (for text) as pretrained encoders.
  • Stage 1 (Fine-Tuning): Each encoder is fine-tuned separately on the in-cabin emotion task to learn modality-specific features.
  • Stage 2 (Cross-Modal Alignment): A contrastive learning module aligns embeddings from different modalities (video, audio, text) into a shared emotional subspace, pulling same-segment embeddings closer and pushing different segments apart.
  • Stage 3 (Ensemble Prediction): Multiple lightweight classifiers are trained on various modality combinations. At inference, their outputs are aggregated via soft voting to enhance robustness against noise and missing modalities.
• Driver State Monitoring: For fatigue, the system uses MediaPipe FaceMesh to extract landmarks and compute metrics like Eye Aspect Ratio (EAR) and PERCLOS. For distraction, it employs CLIP's visual encoder to identify specific actions.

Key Results

Emotion Recognition

• Multimodal Superiority: Experimental results demonstrate that multimodal fusion consistently outperforms unimodal baselines. The full multimodal setting (VAT) achieved the highest performance with 82.25% accuracy and 79.56% F1-score.
• Modality Analysis: Audio was found to be the most informative single modality (73.80% accuracy), while text alone performed poorly (41.69% accuracy) in zero-shot settings, highlighting the necessity of visual and acoustic cues in driving contexts.
• Baseline Performance: CAMEL-emotion significantly outperformed other state-of-the-art models, including fine-tuned versions of Emotion-LLaMA (74.65%) and AffectGPT (73.24%).
• Robustness: The model maintained strong performance under low-light and noisy conditions where single modalities often fail.

Driver State Monitoring

• Fatigue Detection: CAMEL-state achieved 84.58% accuracy, substantially outperforming zero-shot large models like GPT-4o (37.89%) and Qwen2.5-VL-7B (66.08%).
• Distraction Detection: The proposed model achieved 81.06% accuracy, again surpassing general-purpose multimodal models.

Cross-Lingual Evaluation

• The auxiliary English benchmark showed consistent trends with the Chinese dataset but revealed performance drops, particularly in audio-involved settings. This highlights the challenges of cross-lingual multimodal emotion recognition while validating the dataset's utility for future cross-cultural research.

Significance and Claims

The paper claims that InCarEmo addresses critical gaps in the field by providing the first dataset that simultaneously incorporates RGB, IR, audio, and dialogue text specifically for in-cabin conversational settings. Its significance lies in:

1. Comprehensive Foundation: It offers a unified benchmark for multimodal affective understanding, covering diverse lighting, viewpoints, and noise conditions.
2. Practical Applicability: The inclusion of an English benchmark and the proposal of a lightweight, efficient baseline (CAMEL) aim to promote robust, interpretable, and human-centric in-cabin systems suitable for real-world deployment.
3. Safety and Interaction: By enabling the detection of emotions, fatigue, and distraction through conversational and behavioral cues, the dataset supports the development of Advanced Driver-Assistance Systems (ADAS) that can reduce safety risks and foster more empathetic driver-vehicle interactions.

The authors emphasize that the dataset and code are publicly available to facilitate reproducible research in affective computing and intelligent mobility.