Incongruent Positivity: When Miscalibrated Positivity Undermines Online Supportive Conversations

1. Problem Statement

The paper addresses the phenomenon of incongruent positivity in emotionally supportive conversations. While positive language is intended to provide support, it can "misfire" when it is miscalibrated to the user's emotional state. Instead of offering comfort, such responses appear dismissive, minimizing, or unrealistically optimistic, thereby invalidating the user's distress and eroding trust.

The study highlights a critical gap in current Large Language Models (LLMs): while they are increasingly used for emotional support, they often rely on a narrow set of support strategies that fail to align with the intensity of the user's distress, particularly in high-stakes contexts (e.g., grief, severe anxiety). Unlike "toxic positivity" (a broad psychological coping mechanism), this study defines incongruent positivity as a specific conversational mismatch between a user's need for acknowledgment and a response that forces a positive tone.

2. Methodology

Data Collection and Dataset Construction

The authors constructed a new dataset of 1,490 conversation turns derived from Reddit, categorized by emotional intensity:

• Severe Concerns: Topics covering grief and anxiety (sourced from r/grief, r/stress).
• Mild Concerns: Topics covering relationship tension and general advice (sourced from r/situationshipsadvice, r/advice).

For each user post, the dataset includes:

1. Human Responses: The highest-ranked human reply from the subreddit.
2. LLM Responses: Generated using Meta Llama 3.2 1B Instruct with fixed decoding parameters.

Annotation Framework

The dataset was annotated for three primary dimensions:

1. Misguided Positivity Categories (Non-exclusive):
   • Dismissive: Invalidates or overlooks the user's concern (implies overreaction).
   • Minimizing: Downplays emotions via generic or oversimplified advice.
   • Unrealistic: Offers impractical, idealistic solutions detached from context.
   • None: Validating responses that do not exhibit the above traits.
2. Reframing Strategies: Growth, Impermanence, Optimism, Thankfulness, and a newly introduced category, Relatability (sharing similar experiences to foster connection).
3. Emotional Relief: A pairwise comparison (Assistant A vs. Assistant B) to determine which response provided greater perceived emotional relief.

Model Training and Evaluation (RQ3)

To investigate if training on emotional data reduces incongruent positivity, the authors:

• Fine-tuned Mistral-7B and Llama-3.2-3B on two subsets of the Sharma et al. (2020) dataset: Weak Emotion (Level 0) and Strong Emotion (Levels 1–2).
• Developed a Weakly Supervised Multi-Label Detection Framework to classify the generated responses. This ensemble combines DeBERTa and MentalBERT.
• Applied threshold tuning (sigmoid activation with optimized per-label thresholds) to handle class imbalance and improve detection of rare categories (Dismissive, Unrealistic).

3. Key Contributions

1. Conceptual Formalization: Defined and operationalized "incongruent positivity" into a taxonomy of Dismissive, Minimizing, and Unrealistic support, distinguishing it from general toxic positivity.
2. New Dataset: Released a dataset of 1,490 paired human/LLM responses across Mild and Severe contexts, annotated for misguided positivity, reframing strategies, and emotional relief.
3. Detection Framework: Proposed and evaluated a weakly supervised ensemble classifier (DeBERTa + MentalBERT) that outperforms baselines in detecting rare forms of misguided positivity.
4. Empirical Analysis: Provided a comparative analysis of human vs. LLM responses across different emotional intensities and the impact of fine-tuning on emotional alignment.

4. Key Results

Prevalence of Misguided Positivity (RQ1)

• LLMs vs. Humans: LLMs are significantly more prone to generating dismissive responses in Severe contexts (15.4% for LLMs vs. 7.0% for humans).
• Context Sensitivity: Humans show high sensitivity to context, with 76.1% of Severe responses falling into the "None" (no misguided positivity) category. In contrast, LLMs only achieved 20.0% "None" in Severe contexts, frequently resorting to Minimizing (33.0%) or Dismissive tones.
• Reframing Strategies: LLMs rely heavily on Impermanence (e.g., "it will pass") and Optimism, whereas humans utilize Relatability more frequently in Severe contexts to build emotional connection.

Emotional Relief Preferences (RQ2)

• Mild Contexts: LLMs were perceived as providing more emotional relief (65.8%) than humans, suggesting generic positivity is acceptable in low-stakes scenarios.
• Severe Contexts: Humans were strongly preferred (50.1% vs. 1.6% for LLMs). The "No Relief" category was significantly higher for Severe contexts (21.3%), indicating the difficulty of providing resonant support in grief/anxiety scenarios.

Impact of Fine-Tuning (RQ3)

• Strong Emotion Fine-Tuning: Surprisingly, fine-tuning on strong emotional reactions increased the prevalence of Minimizing language (e.g., 62.1% in Mistral-Mild), suggesting models learned to offer surface-level reassurance rather than deep engagement.
• Weak Emotion Fine-Tuning: Training on weak emotional reactions increased the "None" category (neutral/congruent responses), reducing misguided positivity.
• Conclusion: Fine-tuning reduces extreme unrealistic optimism but does not eliminate the tendency to minimize complex distress.

Classification Performance

• The Ensemble (DeBERTa + MentalBERT) achieved the best Macro-F1 score (0.52), significantly outperforming baselines like SVM, LR, and single BERT models.
• The ensemble showed the most balanced error profile, avoiding the over-prediction of "Minimizing" seen in other models.

5. Significance and Implications

• Context-Aware Deployment: The study argues against the blanket use of LLMs for severe emotional support. While LLMs are viable for mild advice, they risk causing harm in high-stakes situations (grief, anxiety) by invalidating user emotions through miscalibrated positivity.
• Beyond Generic Positivity: Effective supportive dialogue systems must move beyond generating "positive" words to achieving emotional alignment. This requires balancing positive affect with genuine acknowledgment of distress.
• Safety and Trust: The findings highlight a safety concern where LLMs may inadvertently amplify distress. The paper suggests implementing "human-in-the-loop" safeguards for high-stakes interactions and developing detection mechanisms for incongruent positivity.
• Future Directions: The authors call for training strategies that go beyond simple emotion classification to include intensity-aware modeling and the integration of human-centered measures of support effectiveness.

In summary, this paper provides empirical evidence that current LLMs struggle to calibrate positivity to emotional intensity, often defaulting to dismissive or minimizing responses in critical situations. It offers a new taxonomy, dataset, and detection framework to guide the development of more empathetic and context-aware conversational AI.