Human-like Affective Cognition in Foundation Models

1. Problem Statement

While recent Large Language Models (LLMs) have demonstrated proficiency in sentiment analysis and facial expression recognition, these tasks are primarily perceptual (recovering a label from a stimulus). True human emotional understanding, known as affective cognition, involves complex reasoning. It requires inferring how individuals subjectively interpret situations based on their beliefs, goals, and expectations (cognitive appraisals) to derive emotions.

The core problem addressed is the lack of a principled framework to evaluate whether foundation models can perform this deeper causal reasoning. Existing benchmarks often lack:

1. A systematic taxonomy of affective inferences.
2. Controlled, synthetic stimuli that isolate specific causal variables (e.g., separating the outcome from the appraisal of that outcome).
3. A comparison against human ground truth that accounts for inter-participant variability.

2. Methodology

A. Theoretical Framework: Causal Appraisal Theory

The authors ground their evaluation in psychological theories of affective cognition (specifically Cognitive Appraisal Theory). They posit that emotions arise from the interaction of:

• Outcomes: The actual event.
• Appraisals: The subjective evaluation of the event (e.g., Goal Congruency, Perceived Control, Safety, Expectedness).
• Emotions: The resulting affective state.
• Expressions: Observable behavioral outputs (facial expressions).

B. Stimulus Generation Pipeline

The authors developed a three-stage procedural generation pipeline to create 1,280 diverse scenarios:

1. Abstract Causal Template: Defined a causal graph where Outcomes and Appraisals cause Emotions, which in turn cause Expressions.
2. Population via LLMs: Used LLMs to generate background stories (e.g., "Amy applying to college") and populate the causal variables with binary values (e.g., Goal Congruent vs. Incongruent; High Control vs. Low Control).
3. Stimulus Composition: Combined these variables into specific inference tasks.
   • Experiments 1a/1b: Text-only scenarios focusing on Goal Congruency/Control and Safety/Expectedness.
   • Experiments 2a/2b: Multimodal scenarios adding Facial Expressions (generated via Unity using Facial Action Units) as an observed variable.

C. Experimental Design

• Inference Tasks: For each scenario, the model (or human) is given three variables and asked to infer the fourth (e.g., Given Outcome + Appraisal + Expression $\rightarrow$ Infer Emotion).
• Models Evaluated: GPT-4 (turbo), Claude-3.5 Sonnet, and Gemini-1.5 Pro.
• Prompting Strategies:
  • 0-shot: Direct question.
  • 0-shot Chain-of-Thought (CoT): "Let's think step by step" prompting to encourage reasoning.
• Human Baseline: 567 native English-speaking participants provided judgments for all 1,280 items. The "ground truth" was defined as the modal human response (majority vote), rather than the theoretical labels assigned during generation.

3. Key Contributions

1. A Principled Evaluation Framework: Introduced a systematic method to generate and benchmark affective reasoning tasks based on causal psychological theory, moving beyond simple sentiment classification.
2. Synthetic Data Generation: Demonstrated a scalable pipeline to create diverse, controlled affective scenarios with known causal structures, allowing for the isolation of specific reasoning dimensions.
3. Human-Model Alignment Benchmark: Established a rigorous comparison between foundation models and human intuitions, using inter-participant agreement as a baseline for "human-level" performance.
4. Multimodal Reasoning Analysis: Evaluated how models integrate textual context with visual facial expressions to infer mental states.

4. Key Results

A. Validation of Stimuli

• Human Agreement: Participants showed high agreement with the majority choice (significantly above chance), validating the stimuli's ability to elicit coherent human judgments.
• Theory vs. Intuition: Inter-participant agreement often exceeded the agreement between participants and the theoretical labels assigned during generation (e.g., 69.38% vs. 41.49% for emotion inference in safety scenarios). This highlights that laypeople's intuitive theories of emotion can diverge from strict psychological taxonomies.

B. Model Performance vs. Humans

• High Alignment: Foundation models (GPT-4, Claude, Gemini) achieved agreement rates with human modal judgments that matched or exceeded inter-participant agreement in several tasks.
  • Example: In inferring outcomes from emotions, models reached 80-90% agreement, often higher than the human inter-rater reliability (76%).
• Task Specificity:
  • Models excelled at inferring Outcomes and Goal Congruency.
  • Models struggled more with Safety and Expectedness appraisals compared to humans.
• Multimodal Integration:
  • Providing facial expressions generally improved emotion inference accuracy by ~8.9 percentage points.
  • However, for outcome inference, some models (e.g., Claude) saw performance drop when expressions were added, suggesting potential difficulty in reconciling conflicting cues.

C. The Role of Reasoning (Chain-of-Thought)

• Significant Improvement: Prompting models to use Chain-of-Thought (CoT) reasoning consistently improved performance across all models and tasks.
• Quantitative Gain: CoT increased agreement with human judgments by an average of 6.9 percentage points.
  • Example: GPT-4's agreement on goal-conduciveness inference jumped from 71.14% (0-shot) to 88.61% (CoT).
• Implication: This suggests that the models possess the latent knowledge for affective reasoning but require explicit reasoning steps to access it effectively.

D. Distributional Understanding

• Beyond predicting the majority answer, models were tested on their ability to capture the full distribution of human responses (using Wasserstein distance). Models performed above chance, indicating they can represent the nuance and variability of human emotional judgment, not just the "average" answer.

5. Significance and Future Directions

• Conceptual Understanding: The results suggest that foundation models have acquired a conceptual understanding of human emotions, capable of reasoning about the causal links between beliefs, goals, and affective states.
• Beyond Perception: The study proves that AI can move beyond "reading" emotions (perception) to "understanding" them (reasoning about mental states).
• Applications: These capabilities are crucial for developing empathetic AI assistants, mental health support tools, and social robots.
• Risks: The authors caution that models potentially outperforming average humans in emotional reasoning could be misused for manipulation or deception, necessitating rigorous ethical guidelines.
• Future Research: The framework is generalizable to a wider range of appraisals (47 identified in literature) and other social cognition tasks. Future work should investigate the mechanistic representation of these capabilities within neural network weights and the role of pre-training vs. alignment.

In conclusion, the paper provides strong evidence that modern foundation models can perform human-like affective cognition, particularly when guided by chain-of-thought reasoning, marking a significant step toward AI that truly understands the human emotional experience.