💬 NLP

HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

The paper introduces HEART, a unified benchmark that directly compares humans and large language models in emotional support dialogues using a science-grounded rubric, revealing that while frontier models approach human levels in empathy and consistency, humans still excel in adaptive reframing and nuanced tone shifts, all while demonstrating strong alignment between human and automated evaluation criteria.

Original authors: Laya Iyer, Kriti Aggarwal, Sanmi Koyejo, Gail Heyman, Desmond C. Ong, Subhabrata Mukherjee

Published 2026-02-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Laya Iyer, Kriti Aggarwal, Sanmi Koyejo, Gail Heyman, Desmond C. Ong, Subhabrata Mukherjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are having a really bad day. You call a friend to vent, and they say, "I'm sorry you're feeling that way." It's nice, but it feels a little generic. Then, you call another friend, and they say, "It sounds like you're feeling overwhelmed because your boss ignored your email again, and that's making you feel invisible. Do you want to talk about how to handle that, or just vent for a minute?"

The second friend "gets it." They aren't just saying the right words; they are listening and adapting to your specific mood.

This paper introduces HEART, a new way to test if Artificial Intelligence (AI) can do that second kind of listening as well as humans can.

Here is the breakdown of what the researchers did, using simple analogies:

1. The Problem: The "Robot" vs. The "Real Friend"

For a long time, we've tested AI on how well it can solve math problems, write code, or summarize news. But can an AI be a good friend when you are sad, angry, or scared?

The Old Way: We used to ask AI, "What emotion is this?" (Like a weatherman saying, "It's raining.")
The New Problem: Real life isn't just about naming the weather; it's about knowing whether to bring an umbrella, a hot chocolate, or just sit in silence with someone. Previous tests didn't check if AI could handle the messy, emotional, back-and-forth of a real conversation.

2. The Solution: The HEART Benchmark

The researchers created a "gym" for testing emotional support. They call it HEART.
Instead of just asking the AI to write a nice sentence, they put it in a multi-turn conversation (like a real phone call) where a person is struggling. They then compare the AI's response side-by-side with a response from a real human.

They judge these conversations using 5 "Muscles" (dimensions):

Human Alignment: Does it sound like a real person, or a robot reading a script?
Empathic Responsiveness: Does it validate your feelings without judging you?
Attunement: Does it notice the tiny details you mentioned earlier? (e.g., "You said your dog was sick, is he okay?")
Resonance: Does it help move the conversation forward with a helpful next step?
Task-Following: Does it stay in its lane? (e.g., An AI shouldn't give medical advice if it's not a doctor).

3. The Big Surprise: AI is "Too Good" at Being Nice

The results were shocking.

The "Polite Robot" Effect: When humans rated the conversations, they often picked the AI's response over the human's response about 47% of the time.
Why? Humans are messy. We get tired, we interrupt, we use awkward phrasing, or we sometimes say the wrong thing. AI is consistently polite, fluent, and never gets frustrated. It sounds like the "perfect listener" on paper.
The Catch: The AI is good at sounding empathetic (the "form"), but humans are still better at being empathetic (the "function"). Humans can handle it when you get angry at them or when you are being difficult. AI tends to just keep saying, "I'm sorry, that sounds tough," even when you are yelling at it.

4. The "Speed vs. Quality" Trade-off

The paper also looked at how fast these AI models are.

The Slow Giants: The smartest, most empathetic-sounding AIs are often very slow. They take several seconds to think before they speak. In a real conversation, a 3-second silence feels awkward and breaks the connection.
The Speedsters: Some models (like the one from Hippocratic AI mentioned in the paper) are incredibly fast (under 0.5 seconds) but still score high on empathy.
The Metaphor: Imagine a therapist who is a genius but takes 5 minutes to answer every question. It's helpful, but it kills the flow. The goal is a therapist who is both a genius and quick to respond.

5. The "Adversarial" Test (The "Bad Day" Test)

The researchers added a special test where the "seeker" was angry, skeptical, or resistant.

Humans: When someone is angry, a good human friend might say, "Hey, I can tell you're really frustrated right now. Let's take a breath." They can handle the tension.
AI: When the seeker got angry, the AI mostly just kept repeating, "I'm sorry you feel that way." It tried to be polite but failed to de-escalate the situation.
Lesson: AI is great at "surface empathy" (being nice), but humans are better at "deep empathy" (navigating conflict and complex emotions).

6. The Verdict: What Does This Mean?

AI is catching up fast: In terms of sounding supportive, AI is now almost as good as the average human.
But it's not a replacement yet: AI lacks the "gut feeling" to know when to push back, when to set boundaries, or how to handle a truly difficult emotional moment.
The Future: We shouldn't just ask, "Is the AI empathetic?" We need to ask, "Is the AI safe and effective?" The paper suggests that while AI can be a great "first responder" for emotional support, we need to be careful not to let it replace human connection, especially for people in crisis.

In a nutshell:
Think of HEART as a driving test for AI's emotional intelligence. The AI has learned to drive very smoothly and politely (better than many humans!), but it still struggles when the road gets bumpy, the other driver is yelling, or the weather turns stormy. We need to teach it how to handle the rough patches before we let it drive us anywhere.

1. Problem Statement

While Large Language Models (LLMs) have achieved significant progress in reasoning and language generation, their ability to provide emotional support in high-stakes, multi-turn conversations remains poorly understood. Existing benchmarks primarily focus on cognitive tasks (classification, reasoning) or single-turn sentiment analysis, failing to capture the interpersonal dimension of dialogue.

The Gap: There is no unified framework to directly compare human and model performance on the same emotional support tasks.
The Challenge: Supportive dialogue requires dynamic skills like reading emotions, adjusting tone, navigating resistance, and maintaining safety, which are not captured by standard metrics like BLEU or single-turn empathy classification.
The Question: Do current frontier LLMs possess the "affective reasoning" necessary to match or surpass human performance in providing empathetic, safe, and effective support?

2. Methodology: The HEART Framework

The authors introduce HEART, a unified benchmark designed to evaluate humans and LLMs side-by-side on multi-turn emotional support conversations.

A. Dataset Construction

Source: Derived from ESConv, a corpus of emotional support dialogues.
Scale: 300 dialogue histories (280 regular, 20 adversarial).
Scenarios: Cover diverse high-stakes situations (grief, academic stress, relationship conflict, exhaustion).
Adversarial Variants: 20 items where the "seeker" expresses anger, skepticism, or resistance to test adaptive skills beyond polite pattern matching.
Generations: Each dialogue is completed by 3 independent human annotators and 16 LLMs (across 5 model families: GPT, Claude, Gemini, Kimi, Llama, and Polaris).

B. Evaluation Protocol

Blinded Pairwise Comparison: Judges (both human and LLM) are presented with two responses (A/B) for the same dialogue context and asked to choose the more supportive one.
Judges:
- Human Judges: 15 annotators who were not provided with the evaluation rubric to ensure natural, intuitive judgment.
- LLM Judges: An ensemble of models (GPT, Claude, Gemini) provided with the structured rubric to test evaluative alignment.
Scoring: A Bradley-Terry model is used to aggregate pairwise preferences into Elo-style ratings for ranking systems.

C. The HEART Rubric (5 Dimensions)

The evaluation is grounded in communication science and counseling psychology, measuring five distinct axes:

Human Alignment (H): Does the response sound natural, thoughtful, and human-like in tone and flow?
Empathic Responsiveness (E): Does the supporter validate feelings, convey understanding, and avoid judgment?
Attunement (A): Does the response track specific details and emotional signals (e.g., naming the feeling) rather than offering generic reassurance?
Resonance (R): Does the response move the conversation forward helpfully (e.g., concrete next steps, clarifying needs)?
Task-Following (T): Does the response stay within safety/role boundaries and address the seeker's goals?

3. Key Contributions

First Human-Model Direct Comparison: HEART is the first framework to place humans and LLMs on the same multi-turn, emotionally complex tasks, enabling rigorous comparison of supportive behavior.
Behavioral Taxonomy: It operationalizes "affective reasoning" into five measurable, validated interpersonal dimensions, moving beyond surface-level sentiment.
Adversarial Testing: The inclusion of "resistance" scenarios tests a model's ability to handle pushback, a critical failure point for current AI.
Evaluator Alignment Analysis: The study uniquely analyzes the agreement between human judges and LLM-as-judge evaluators to understand how models internalize social preferences.
Latency vs. Quality Trade-off: The benchmark introduces a critical analysis of Time-to-First-Token (TTFT) latency, linking conversational quality to real-time voice agent feasibility.

4. Key Results

A. Performance: LLMs vs. Humans

Perceived Empathy: Frontier LLMs often surpass average human responses in perceived empathy and consistency. Human raters preferred model responses in 46.8% of cases, compared to 35.8% for human responses (17.4% ties).
Evaluator Agreement: Human and LLM judges agree on the "better" response in ~78.7% of cases, closely matching inter-human agreement (79.5%).
Rationale Alignment: When humans and LLMs agree on a winner, they cite similar justifications (e.g., "names the feeling," "offers a next step") in 68% of cases, suggesting convergence in what constitutes supportive quality.

B. Strengths and Weaknesses

Model Strengths: Models excel in stylistic coherence, fluency, and emotional stability. They consistently avoid the hesitations, bluntness, or emotional fatigue seen in human dialogue.
Human Strengths: Humans outperform models in adaptive reframing, naming tension, and nuanced tone shifts, particularly in adversarial (resistant) turns. Humans use a more diverse and balanced repertoire of counseling strategies, whereas models rely heavily on a narrow subset (reassurance, acknowledgment).
The "Surface vs. Depth" Gap: Models achieve high scores through "surface empathy" (politeness, fluency), while humans demonstrate "experienced empathy" (contextual judgment, boundary setting).

C. Latency and Real-Time Viability

The Latency Trade-off: Generally, HEART Elo scores correlate with latency (slower, larger models perform better). Top frontier models (e.g., GPT-5, Claude 4.5) have TTFTs of several seconds, which disrupts natural turn-taking in voice interactions.
The Exception (Polaris 4): Hippocratic AI's Polaris 4 achieves a high Elo (1604.0) with a median TTFT of ~400ms. It is the only model operating in the sub-500ms regime while scoring above the human baseline, making it uniquely suitable for real-time healthcare voice agents.

D. General Reasoning vs. Affective Reasoning

While HEART performance correlates with general intelligence benchmarks (Pearson $r=0.70$ ), the relationship is not deterministic. Models with similar general intelligence scores can diverge significantly on HEART due to differences in tone control and instruction tuning. This suggests affective reasoning requires specific domain alignment, not just general capability.

5. Significance and Implications

Redefining Empathy: The paper argues that empathy in AI is currently a behavioral approximation (mimicking the form of empathy) rather than a deep contextual judgment.
Safety and Ethics: The finding that models are often preferred over humans raises ethical concerns about over-reliance on AI for mental health support. Models may provide "safe-sounding" but ineffective advice, potentially blurring the line between instrumental empathy and affective deception.
Design Guidelines: For AI to be truly helpful, it must move beyond generic warmth to include calibrated challenge, boundary setting, and adaptive strategy.
Future Directions: The authors call for longitudinal studies to measure actual user well-being outcomes, not just perceived empathy, and emphasize the need for cross-cultural validation of the benchmark.

In conclusion, HEART provides a principled foundation for measuring the "interpersonal intelligence" of AI, revealing that while frontier models have mastered the language of empathy, they still lack the adaptive reasoning of human supporters, particularly in complex or resistant interactions.