Technical Summary: Empathy Applicability Modeling for General Health Queries

Problem Statement

Large Language Models (LLMs) are increasingly integrated into clinical workflows, yet they frequently lack clinical empathy, a critical component of effective doctor-patient communication. While existing Natural Language Processing (NLP) frameworks can reactively label empathy in responses (e.g., EPITOME, Online Empathy), they offer limited support for anticipatory modeling—determining before a response is generated whether a patient query necessitates an empathetic reaction.

Current approaches face several limitations:

• Reactive Nature: Most frameworks assess empathy post-hoc, providing no guidance to clinicians or systems while composing a response.
• Context Blindness: Systems often over-rely on surface lexical cues, misclassifying generic phrases as empathetic in nonsensical contexts (false-positive rates >95% in some cases).
• Dimensionality: Prior work often over-weights emotional aspects while neglecting cognitive empathy (understanding), or focuses on stance shifts in synchronous dialogues rather than the specific needs (cognitive clarification vs. emotional warmth) in single-turn, asynchronous general health queries.

The core challenge addressed is: How can we systematically model the applicability of empathy to recognize specific clinical and linguistic cues that necessitate an emotional or interpretive response in general health queries?

Methodology

1. The Empathy Applicability Framework (EAF)

The authors introduce EAF, a theory-driven approach that classifies patient queries based on the applicability of two empathy dimensions, adapted from EPITOME and Patient-Centered Care (PCC) literature:

• Emotional Reactions (EA): Expressions of warmth, compassion, or concern.
• Interpretations (IA): Communication of understanding regarding the patient's feelings and/or experiences.

For each dimension, queries are labeled as Applicable or Not Applicable based on specific cues:

• Applicable Cues: Severe negative emotion, inferred negative states, seriousness of symptoms, concern for relations, expression of feelings, distressing uncertainty, and symptoms affecting daily life.
• Not Applicable Cues: Routine health management, purely factual queries, neutral symptom descriptions, and hypothetical queries without emotional concern.

2. Dataset Curation and Annotation

• Source: 9,500 patient queries sampled from public datasets (HealthCareMagic and iCliniq).
• Annotation Strategy:
  • Human Annotators: Two lay annotators (non-clinicians) with high English proficiency were recruited to capture the patient's perspective. They underwent a three-stage training process involving 200 queries.
  • LLM Annotation: GPT-4o was used as an expert annotator via contrastive prompting.
  • Subsets:
    • Human-GPT Subset: 1,300 queries dual-annotated by humans and GPT-4o (5 passes for GPT, majority vote).
    • Autonomous Subset: 8,000 queries annotated solely by GPT-4o (single pass).
• Task: Annotators identified the applicability (Applicable/Not Applicable) and selected the best-fitting subcategory for each dimension.

3. Modeling Approach

The task was framed as two independent binary classification problems (predicting applicability for EA and IA).

• Models: RoBERTa-base classifiers were fine-tuned.
• Training Sets:
  • Human Set: Trained on the 1,300 queries where human annotators reached consensus.
  • Autonomous Set: Trained entirely on the 8,000 GPT-labeled queries.
• Baselines: Comparisons were made against Random Guessing, "Always Applicable/Not Applicable" heuristics, classical text classifiers (Logistic Regression, Linear SVM with TF-IDF), and a zero-shot LLM baseline (o1) without the EAF framework.

Key Results

1. Reliability and Consistency

• Human Agreement: Human annotators achieved moderate agreement (Cohen's $\kappa \approx 0.46$), consistent with typical empathy annotation tasks.
• Human-GPT Alignment: GPT-4o showed strong alignment with human consensus on the subset where humans agreed ($\kappa > 0.6$ for both dimensions), with raw agreement around 80%.
• Conceptual Alignment: UpSet plot analysis revealed that when humans and GPT agreed on the binary label, they often relied on similar subcategory rationales, particularly for clear cues like "Severe Emotion" or "Factual Queries."

2. Predictive Validity

Classifiers trained on EAF-labeled data significantly outperformed all baselines:

• Human-Supervised Models: The RoBERTa model achieved a Macro-F1 of 0.92 for Emotional Reactions and 0.87 for Interpretations, surpassing classical baselines (LR/SVM ~0.80–0.84) and zero-shot LLMs.
• Autonomous Models: Models trained solely on GPT labels also performed well (Macro-F1 ~0.85 for EA, ~0.78 for IA) when tested on the human-consensus set, demonstrating that EAF patterns are learnable even from synthetic labels, though with a slight performance drop due to label noise.

3. Systematic Challenges

Error analysis identified three persistent challenges in operationalizing anticipatory empathy:

1. Subjectivity in Implied Distress: Divergence in labeling "Inferred Negative State" and "Distressing Uncertainty," where one annotator might infer distress while another views the query as factual.
2. Clinical-Severity Ambiguity: GPT often flagged "Serious Symptoms" requiring empathy where humans did not (e.g., overgeneralizing vivid but non-life-threatening pain), though it also correctly identified severe conditions (e.g., post-transplant complications) that lay annotators missed.
3. Contextual Hardship & Cultural Bias: GPT frequently over-applied tags for "Symptoms Emotional Impact" and "Context Sharing," potentially equating physical discomfort with emotional distress. This suggests a Western-centric bias in GPT-4o compared to the South Asian human annotators.

Significance and Contributions

The paper claims three primary contributions to the field of clinical empathy in NLP:

1. Framework Design: The introduction of the Empathy Applicability Framework (EAF), which shifts the paradigm from reactive response scoring to anticipatory modeling. By analyzing the patient query before response generation, EAF enables systems to detect specific needs for emotional or interpretive support.
2. Benchmark Establishment: The release of a novel, dual-annotated benchmark of 1,300 patient queries. This dataset demonstrates the reliability of EAF labels and provides a standard for evaluating anticipatory empathy models.
3. Operationalization Insights: A systematic analysis of the challenges in applying anticipatory empathy, highlighting the need for multi-annotator modeling, clinician-in-the-loop calibration, and culturally diverse annotation strategies to address subjectivity, clinical ambiguity, and cultural bias.

The authors emphasize that EAF is designed to augment, not replace, clinician judgment. They advocate for a "human-in-the-loop" pipeline where the framework provides transparent rationales to support clinicians in detecting empathic opportunities, thereby mitigating risks of artificial empathy (e.g., uncanny valley effects or false intimacy) while enhancing asynchronous healthcare communication.