Technical Summary: Evaluating Multimodal Emotion Recognition in Proactive Conversational Agents

Problem Statement

Current emotion detection systems for Socially Interactive Agents (SIAs) predominantly rely on pre-recorded datasets, theoretical simulations, or passive observation. There is a significant gap in research regarding multimodal emotion detection within dynamic, real-time conversations driven by Generative AI. Specifically, existing literature fails to adequately address how proactive, AI-generated dialogues influence a user's internal affective states and facial expressions during live exchanges. Furthermore, the reliability of automated visual cues (facial recognition) versus linguistic analysis in unscripted human-agent interactions remains under-explored, particularly regarding the discrepancy between a user's internal state and their external "poker face."

Methodology

The study presents the design and empirical evaluation of a proactive SIA powered by Generative AI. The research employed a within-subjects experimental design involving 20 participants (ages 22–90) in a controlled laboratory setting.

System Architecture

The system integrates two core emotion detection channels:

1. Computer Vision Module: Utilizes SilNet, a computer vision framework optimized for Facial Expression Recognition (FER), to analyze real-time facial cues.
2. Linguistic Analysis Module: Leverages Generative AI (OpenAI API/GPT-4) to perform semantic sentiment analysis. The system analyzes the user's speech every three interaction turns to evaluate the emotional trajectory of the discourse rather than isolated words.

Experimental Procedure

The evaluation consisted of two distinct dialogue rounds with varying roles (Friend, Expert, Psychologist) and topics (Family, Nature, Animals, Random).

• Data Collection: The study triangulated three data sources:
  1. Automated Visual Data: Real-time FER classifications.
  2. Automated Linguistic Data: Generative AI sentiment analysis.
  3. Subjective Ground Truth: User self-reports using a 2D Valence-Arousal Matrix (Russell's Circumplex Model) and post-dialogue questionnaires (8-item and 20-item scales).
• Goal: To compare the automated detection methods against the users' self-reported "ground truth" to determine detection efficacy and identify interaction patterns.

Key Results

1. The "Poker Face" Effect and Visual Discrepancy

A significant discrepancy was found between automated visual cues and actual internal emotional states.

• Visual Failure: The facial recognition module demonstrated low accuracy (23.8%), performing only slightly better than random chance. It exhibited a strong bias toward detecting negative emotions (specifically "Disgust" and "Anger"), which accounted for 62.5% of its classifications.
• The Phenomenon: Users consistently displayed serious, concentrated facial expressions ("poker face") even when experiencing positive emotions (e.g., Happiness, Calmness). The system misinterpreted this state of deep concentration or engagement as negative affect (rejection or hostility).
• User State: Self-reports indicated users predominantly felt positive, calm, or attentive (low arousal, high valence), contradicting the visual system's negative classification.

2. Superiority of Linguistic Analysis

The Generative AI-based linguistic analysis proved significantly more reliable than visual recognition.

• Accuracy: The linguistic module achieved an accuracy of 45%, substantially outperforming the visual modality.
• Strengths: It successfully identified positive emotions like "Happiness" and "Sadness" by contextualizing verbal expressions.
• Limitations: The module struggled with the "lexical fallacy," occasionally misinterpreting negative vocabulary used in past anecdotes (e.g., describing a scary event) as the user's current emotional state. It also had difficulty detecting low-arousal states like "Attention" or "Calmness."

3. Interaction Dynamics and Proactivity

• Elicitation: The SIA effectively elicited specific emotions by adapting conversational themes (e.g., animals for joy, environmental degradation for sadness/anger) and employing structured linguistic patterns (empathetic or humorous language).
• User Experience: Participants reported high satisfaction, empathy, and engagement. However, instances of uncalibrated proactivity (e.g., forced jokes or mismatched emotional tones) occasionally led to user disengagement, short answers, and a perception of artificiality.
• Calibration: Users underwent a "calibration" phase; perception of the agent's empathy and understanding improved from the first to the second dialogue session.

Key Contributions

1. Empirical Validation of the "Poker Face" in HCI: The study provides concrete evidence that in human-agent interactions, neutral or serious facial expressions often signify high engagement or concentration rather than negative affect, rendering standard FER unreliable in this context.
2. Modality Comparison: It establishes that for proactive, generative AI dialogues, linguistic context is a more reliable indicator of user emotion than facial cues, challenging the prevailing assumption that multimodal fusion (face + text) is always superior without context-specific calibration.
3. Design Framework for Proactive SIAs: The research highlights the necessity of refining SIAs to dynamically adapt to users' emotional evolution, relying on deep linguistic context rather than superficial visual cues to foster natural interactions.

Significance and Claims

The paper claims that its findings highlight a critical limitation in current SIA design: the over-reliance on visual cues which can be misleading in human-screen interactions. The authors argue that to humanize conversational agents, systems must:

• Prioritize linguistic modality over visual recognition to counteract the "facial neutrality" common in these interactions.
• Calibrate for the "poker face" by recognizing that neutral expressions often mask high engagement.
• Implement "affective probing" to resolve discrepancies between a serious face and positive discourse.

The study concludes that while Generative AI can effectively simulate social presence and elicit emotions, the "expressive gap" between internal states and external behaviors requires a shift in detection strategies. Future work must move beyond simple command-following to genuine conversational companionship by accounting for low-arousal states and the nuances of human language, rather than relying on potentially misleading facial data.