Affect-Aware Human-Agent Conversations: Towards Empathic LLM Interaction

Technical Summary: Affect-Aware Human-Agent Conversations

Problem Statement
Large Language Models (LLMs) possess conversational capabilities but lack authentic emotional experience, relying instead on probabilistic pattern recognition to simulate empathy. This creates a "compassion illusion," where algorithmic recognition of emotional cues is mistaken for genuine resonance. Current human-AI interactions are predominantly text-based, excluding non-verbal affective cues that are central to human interpretation. Consequently, LLM responses often lack "relational grounding," depending solely on explicit textual input rather than the rich, situated context of human communication. While previous work introduced "Empathic Prompting" as a proof of concept, it remained exploratory and limited to rapid pilot implementations without robust real-time integration or comprehensive user validation.

Methodology
The study proposes and evaluates Empathic Extended Prompting, a real-time affect-aware framework designed to enrich human-LLM conversations with facial affective descriptors.

• System Architecture: The system utilizes a client-middleware-server architecture.

  • Client: Captures user facial expressions via a camera stream.
  • Middleware: Processes video using Noldus FaceReader to extract high-level affective descriptors (dominant emotion, intensity, valence, arousal) rather than raw Action Units. A custom middleware filters, synchronizes, and serializes these descriptors into a compact JSON-like format, storing them in an Emotional Parameters Queue (EPQ) to maintain short-term memory of the user's state.
  • Server: Upon user text submission, the server retrieves the most recent affective samples. A 3-second temporal window applies majority voting for the dominant emotion and averages intensity, valence, and arousal. These numerical values are mapped to discrete linguistic labels (e.g., "HIGH" intensity, "POSITIVE" valence) to ensure interpretability by the LLM.
  • Prompt Construction: The system constructs an augmented user turn ($\tilde{M}_t$) by prepending the real-time affective state ($F^t_e$) and a session-level affective prior ($P^t_e$, derived from a temporal histogram of the priming phase) to the user's text ($M_t$). This augmented context is combined with a persistent system prompt defining empathic interaction rules before being sent to the LLM.

• Experimental Design: A within-subjects study was conducted with N = 20 participants.

  • Conditions: Participants interacted with the LLM under two conditions: an Experimental condition (affect-aware) and a Control condition (empathic text-only baseline).
  • Procedure: The session began with an emotional priming phase using validated video clips (The Lion King for sadness, Wall-E for joy) to induce specific affective states. Participants then engaged in a 5-minute semi-structured conversation about their emotional experience. The order of conditions and video stimuli was counterbalanced.
  • Model: The deepseek-r1:32b model, hosted locally via OLLAMA, served as the conversational backbone to ensure privacy and consistency.
  • Measures: Evaluation included the System Usability Scale (SUS), Perceived Empathy of Technology Scale (PETS), Networked Minds Measure of Social Presence, and the Godspeed Questionnaire Series (focusing on Likability, Competence, and Safety). A forced-choice ranking assessed explicit preference.

Key Contributions

1. Real-Time Affect Integration: The study implements a functional pipeline that continuously extracts, processes, and injects facial affective descriptors into the LLM context without exposing raw biometric data to the model.
2. Refined Prompting Strategy: Unlike previous approaches that merely appended cues, this framework grounds the system prompt in an explicit empathic interaction model. Affective cues inform the model's conversational stance (e.g., pacing based on arousal, masking text-face discrepancies) rather than acting as simple metadata.
3. Empirical Validation: The paper provides an end-user validation study comparing the affect-aware system against a text-only baseline, specifically investigating whether real-time affective context improves perceived interaction qualities without inducing negative effects like the "uncanny valley" or reduced usability.

Results

• Perceived Qualities: The affect-aware condition showed higher descriptive scores across key interaction dimensions. Notable improvements were observed in Godspeed Competence ($\Delta = +0.31$), PETS Understanding and Trust ($\Delta = +0.21$), PETS Emotional Responsiveness ($\Delta = +0.16$), and Godspeed Likability ($\Delta = +0.13$).
• Usability and Safety: Usability scores (SUS) remained high and acceptable in both conditions (Control: 81.25, Experimental: 79.38), indicating that the addition of affective processing did not degrade the interface experience. Safety perceptions remained stable between conditions.
• Statistical Significance: At the macro-construct level, paired Wilcoxon signed-rank tests did not reveal statistically significant differences between conditions, likely due to the ceiling effects of modern, instruction-tuned LLMs which already produce supportive responses. However, the descriptive trends consistently favored the affect-aware condition in terms of emotional intelligence and mood responsiveness.
• Negative Effects: The study found no evidence that injecting real-time facial signals elicited negative interactional effects, reduced perceived safety, or created uncanny-like perceptions.

Significance and Claims
The paper claims that integrating real-time affective context can add relational grounding to LLM conversations, shifting the paradigm from "AI-as-a-tool" to "AI-as-a-collaborator." The authors emphasize that the goal is not to make LLMs genuinely empathic (which is impossible given their probabilistic nature) but to improve the user's perception of empathic attunement. By adapting the conversational stance to continuous affective feedback, the system supports a more situated interaction. The findings suggest that multimodal affective grounding enhances perceived emotional intelligence and responsiveness without compromising usability or triggering negative psychological reactions, offering a viable path for more naturalistic human-agent collaboration.