Technical Summary: CAREBench

Problem Statement

Emotion understanding is a critical capability for Large Language Models (LLMs) in affective computing and human-AI interaction. However, existing evaluation paradigms predominantly rely on discrete emotion label prediction, which fails to capture the cognitive processes underlying emotion generation. While recent work has turned to appraisal theory to ground emotion understanding, existing appraisal-based datasets suffer from three key limitations:

1. Lack of Reasoning: They treat appraisal dimension ratings as the sole representation of cognitive appraisal, ignoring the inferential process over mental states (beliefs, desires) that precedes these ratings.
2. Simplistic Emotion Modeling: They rely on single emotion labels, failing to capture the mixed and co-occurring nature of real-world emotional experiences.
3. Perspective Bias: Most datasets provide only third-person annotations, making it difficult to analyze how perspective differences between experiencers and observers influence judgments.

Consequently, it remains unclear whether models truly understand the experiencer's mental states or merely learn superficial mappings from situations to ratings.

Methodology

The CAREBench Dataset

The authors introduce CAREBench, a benchmark designed to evaluate Cognitive Appraisal Reasoning in Emotion understanding. The dataset is grounded in appraisal theory and constructed through a two-phase data collection process involving 1,000 real-world emotional narratives:

• Phase 1 (First-Person): Participants recalled emotionally significant events. Using a "micro-narratives" framework assisted by an LLM, they provided open-ended reasoning for five core appraisal dimensions (relevance, certainty, congruence, control, accountability), hierarchical ratings for 22 sub-dimensions, and multi-label emotion annotations (intensity and labels).
• Phase 2 (Third-Person): Independent annotators read the narratives and performed the same three-layer annotation task without access to the first-person responses.

Key Features:

• Inferential Chain: The dataset establishes a complete $s \to r \to a \to e$ chain: Situation ($s$) $\to$ Reasoning ($r$) $\to$ Appraisal Ratings ($a$) $\to$ Emotion ($e$).
• Dual Perspective: Annotations exist for both the first-person experiencer and multiple third-person observers.
• Multi-Label & Intensity: Emotions are annotated with intensity scores and multiple co-occurring labels.
• Subjectivity Data: Annotator demographics and Big Five personality traits are collected to analyze subjective heterogeneity.

Evaluation Framework

The authors propose a process-level evaluation framework covering the full inferential chain, moving beyond single-step prediction. Six LLMs were evaluated (GPT-5.2, Claude-Sonnet-4.6, Llama 3.1, Qwen 3.5, PsyLLM, and EmoLLaMa) across four Research Questions (RQs):

1. RQ1 (Chain Performance): How well do models perform each step ($s \to r$, $s \to a$, $s \to e$)?
2. RQ2 (Reasoning Utility): Does injecting appraisal reasoning ($r$) or ratings ($a$) improve downstream prediction?
3. RQ3 (Sensitivity): Are models sensitive to structured appraisal interventions (e.g., swapping first-person reasoning for third-person)?
4. RQ4 (Subjectivity): Can models capture the distributional heterogeneity of human appraisal judgments?

Metrics included BLEU/ROUGE for reasoning, RMSE for ratings/intensity, F1 scores for labels, and Wasserstein distance/Spearman correlation for subjectivity modeling.

Key Results

1. Performance Disparities Across the Chain

• Downstream vs. Upstream: Stronger models (e.g., Claude, GPT) match or surpass human third-person observers on appraisal ratings and negative emotion label identification. However, they consistently fall short on appraisal reasoning generation and positive emotion identification.
• Positive Emotion Gap: Models exhibit significant imbalance between precision and recall for positive emotions, leading to substantially lower F1 scores compared to humans.
• Dissociation: Domain-specific fine-tuned models (PsyLLM, EmoLLaMa) showed advantages in specific upstream tasks (e.g., appraisal reasoning metrics) but did not translate these gains to downstream emotion label selection, suggesting that strong downstream performance does not imply correct appraisal-mediated reasoning.

2. The Role of Reasoning

• Human Reasoning: Injecting first-person appraisal reasoning text significantly improved appraisal prediction and emotion recognition (particularly recall) across models.
• Model Reasoning: When models generated their own reasoning ($\hat{r}$) to inject back into the prompt, performance often degraded. Qualitative analysis revealed that models tend to use exaggerated emotional language in self-generated reasoning, introducing noise that propagates to downstream errors.

3. Sensitivity to Interventions

• Models showed varying degrees of sensitivity to structured appraisal interventions. Stronger models (Claude, GPT, Qwen) exhibited moderate-to-strong positive correlations between intervention-induced changes in reasoning and changes in predicted appraisals/emotions, aligning with appraisal theory.
• However, some models with reasonable downstream performance (e.g., Llama, PsyLLM) showed weak directional sensitivity, indicating they may rely on associative patterns rather than genuine appraisal reasoning.

4. Modeling Subjectivity

• While injecting core appraisal reasoning helped models approximate human appraisal distributions, current models have not internalized the mechanisms needed to capture inter-observer disagreement autonomously.
• Models failed to reproduce the variance in human judgments, suggesting they cannot yet model the distributional heterogeneity inherent in human emotion understanding.

Significance and Claims

The paper claims that CAREBench provides a foundation for a more diagnostically informative evaluation of LLMs' affective cognitive capabilities. The primary significance lies in the finding that downstream emotion prediction metrics may overestimate true emotion understanding. By evaluating the full inferential chain, the authors reveal capability gaps—specifically in reasoning generation and positive emotion recognition—that single-step evaluation paradigms miss.

The work suggests that current LLMs have not yet acquired the cognitive capacities required for fine-grained emotion differentiation, particularly those involving complex causal attribution (control, accountability) and positive emotional states. The framework exposes the blind spots of relying solely on output accuracy, offering a methodological basis for developing more cognitively interpretable affective models.

Limitations

The authors note several limitations:

• Cultural Scope: Data is limited to English-speaking participants in the US and UK, potentially limiting generalizability to other cultures.
• Narrative Imbalance: Self-selection of events led to a skew toward negative emotions.
• Annotation Cost: The third-person subset (500 narratives) limits the precision of distributional estimates for subjective variation.
• LLM Assistance: The use of LLMs to assist in narrative composition, while mitigated by review, introduces a risk of incorporating extraneous details.