TEMPER: Testing Emotional Perturbation in Quantitative Reasoning

1. Problem Statement

Large Language Models (LLMs) are typically trained and evaluated on quantitative reasoning tasks (e.g., math word problems) presented in clean, emotionally neutral language. However, real-world user queries often contain emotional framing (frustration, urgency, enthusiasm, or disgust).

The core research question is: Does emotional framing alone degrade reasoning performance, even when all numerical content, logical relationships, and mathematical constraints are perfectly preserved?

Prior studies on LLM robustness have focused on formatting changes, irrelevant context, or paraphrasing, but few have isolated emotional tone as a variable. Existing benchmarks assume neutral input, potentially overestimating model robustness in real-world scenarios where users express emotion.

2. Methodology

The authors developed a controlled framework to generate emotional variants of math problems without altering the underlying math.

A. Teacher-Student Emotion Translation Framework

To generate high-quality emotional translations while preserving mathematical fidelity, the authors introduced a novel Teacher-Student architecture:

• Student Model: A Llama 3.1-8B-Instruct model fine-tuned with LoRA (Low-Rank Adaptation) to translate text.
• Teacher Model: A frozen DistilRoBERTa classifier (pretrained on GoEmotions) that provides auxiliary supervision.
  • The teacher is further trained on 72k emotional math samples to align with the target domain.
  • It provides two supervision signals: a 7-class categorical output (emotion type) and a 100-dimensional latent representation (capturing intensity and style).
• Training Objective: The student minimizes a combined loss:
  $$L = L_{CE} + \lambda \cdot L_{aux}$$
  Where $L_{CE}$ is the standard cross-entropy loss for fluent generation, and $L_{aux}$ aligns the student's hidden states with the teacher's emotion space. The hyperparameter $\lambda$ controls the trade-off between emotional intensity and mathematical fidelity.
• Variants: Five translator variants were created by crossing two supervision types (EMO7 and EMO100) with different $\lambda$ weights, plus a CE-only baseline.

B. Benchmark Construction: TEMPER-5400

• Data Source: Problems were sampled from GSM8K, MultiArith, and ARC-Challenge.
• Process:
  1. The ensemble of five translators generated emotional variants in six Ekman emotions (Anger, Disgust, Fear, Joy, Sadness, Surprise).
  2. Semantic Filtering: A rigorous pipeline (automated checks, LLM-as-a-judge, and human annotation) verified that numbers, quantifiers, and relationships were preserved.
  3. Neutralization: The same translators were used to "neutralize" the emotional variants back to a neutral tone to test recovery.
• Result: A dataset of 5,400 verified emotion–neutral pairs (900 problems $\times$ 6 emotions) comprising 10,800 total translations.

C. Evaluation Setup

• Models: 18 instruction-tuned models ranging from 1B parameters (Tiny) to frontier scale (GPT-5, o3, DeepSeek-V3).
• Conditions: Evaluated under Base prompting (direct answer) and Zero-shot Chain-of-Thought (CoT).
• Controls:
  • Non-emotional paraphrases: Generated using DeepSeek-V3 with strict constraints to match the length and complexity of emotional versions but without emotional content.
  • Neutralization: Testing if converting emotional text back to neutral recovers performance.

3. Key Contributions

1. Controlled Emotion Translation Framework: A novel teacher-student approach that allows for intensity-controllable emotional rewriting while mathematically preserving the source problem.
2. TEMPER-5400 Benchmark: A large-scale, semantically verified dataset of 5,400 emotion-neutral pairs across three reasoning domains.
3. Empirical Evidence of Fragility: Demonstration that emotional framing degrades reasoning accuracy by 2–10% across 18 models, even when numerical content is identical.
4. Neutralization as Mitigation: Evidence that neutralizing emotional text recovers ~70% of the lost performance, proving the degradation is stylistic rather than due to content corruption.
5. Representational Analysis: Hidden-state analysis showing emotional text shifts model representations 3–4$\times$ further from the original than neutral paraphrases, indicating representational interference.

4. Key Results

A. Accuracy Degradation

• Magnitude: Emotional framing reduces accuracy by 2–10% depending on the model and dataset.
  • ARC-Challenge: Highest drop (~6.0%), likely due to emotional language interfering with multiple-choice option discrimination.
  • GSM8K: Mean drop of 3.9% (Base) and 6.1% (Base without CoT).
  • MultiArith: Smallest drop (2.0%) due to ceiling effects.
• Model Scale: Larger models (70B+, Frontier APIs) are more robust but not immune, still suffering 1–3% drops.
• Prompting Strategy: The degradation persists under Chain-of-Thought (CoT) prompting, though it is slightly less severe than in Base prompting.

B. Emotion-Specific Effects

• Most Disruptive: Disgust (mean drop 5.9%, peak 10.0% on ARC) and Fear (4.3%).
• Least Disruptive: Joy (1.9%) and Surprise (2.3%).
• Intensity Correlation: Higher emotional intensity (measured by classifier confidence) correlates with higher failure rates.

C. Failure Patterns

Analysis of reasoning chains revealed three dominant failure modes caused by emotional framing:

1. Constraint Degradation (49%): Emotional elaboration softens or reinterprets mathematical constraints (e.g., ignoring "each" or "per").
2. Attention Competition (30%): Emotional language distracts the model from key numerical elements (e.g., computing a one-way trip instead of round-trip). This is 6.7$\times$ more frequent in emotional text than in neutral paraphrases.
3. Premature Completion (8.6%): Emotional framing foregrounds the target quantity, causing the model to stop reasoning early.

D. Control Experiments

• Non-emotional Paraphrases: Length-matched, non-emotional paraphrases caused no systematic accuracy change (~1% difference), confirming that the degradation is specific to emotional content, not just rewording or increased length.
• Neutralization Recovery: Converting emotional text back to neutral recovered ~70% of the errors (up to 81% on MultiArith), confirming the issue is reversible and tied to style.

E. Representational Evidence

• Cosine Distance: Emotional variants shifted final-layer hidden states 3–4$\times$ further from the original than neutralized or paraphrased versions.
• Linear Probes: A linear classifier could separate emotional text from original/neutral text with near-perfect accuracy (F1 $\approx$ 1.0), suggesting emotional framing creates distinct, structured regions in the representation space that interfere with reasoning.

5. Significance and Implications

• Robustness Gap: Current benchmarks (GSM8K, etc.) significantly overestimate LLM robustness because they ignore the "emotional wrapper" common in real-world interactions.
• Deployment Risk: In real-world applications (e.g., customer support, education), users expressing frustration or urgency may receive incorrect answers even if the math is simple.
• Mitigation Strategy: The paper proposes neutralization as a lightweight, inference-time defense. A small neutralizer model can strip emotional noise before the reasoning model processes the query, recovering most lost accuracy without retraining the main model.
• General Framework: The teacher-student translation method is generalizable to other stylistic perturbations (e.g., formality, sarcasm, politeness), providing a blueprint for isolating style variables from content in NLP evaluation.

In conclusion, TEMPER establishes that emotional framing is a distinct and significant axis of fragility for LLMs, causing measurable reasoning degradation through representational interference and attention distraction, even when the mathematical logic remains untouched.