Technical Summary: EmoDistill

Problem Statement

Post-trained Large Language Models (LLMs) are typically optimized via Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) to be safe, polite, and accommodating. While effective for general assistance, this alignment creates a strategic vulnerability in adversarial negotiation. In such settings, emotionally framed language can inadvertently steer agents toward conceding value to a counterparty's interests.

Current approaches to emotion-aware negotiation face two primary limitations:

1. Online Learning Costs: Methods relying on online reinforcement learning (e.g., PPO) are computationally expensive and unstable for multi-turn agent-to-agent interactions, as they require costly API rollouts and struggle with sparse trajectory-level feedback.
2. Decoupled Strategy and Expression: Existing methods often treat emotion selection as a decision variable but leave the utterance generator fixed. Consequently, a selected emotion (e.g., "anger") may still be realized through vague politeness or premature concession because the model lacks the specific skill to express that emotion strategically within the bargaining context.

The core challenge is to learn emotional negotiation skills—binding dialogue states, emotional stances, concrete utterances, and judge feedback into reusable bargaining moves—without incurring the cost of online training.

Methodology: EmoDistill

The authors propose EmoDistill, an offline framework that distills emotional negotiation skills from precomputed LLM-vs-LLM trajectories into a smaller language model (SLM). The framework decomposes the problem into two distinct components: Emotion Selection and Emotional Expression.

1. Offline Trajectory Dataset and Signal Design

The system begins by constructing a dataset of $N=80$ scenarios $\times$ $M=100$ random emotion-sequence rollouts per domain (totaling 8,000 trajectories). Each trajectory is annotated with two complementary signals:

• Objective Signal ($R(\tau)$): A trajectory-level return based on bargaining dynamics (e.g., closing the gap between the counterparty's anchor and the focal agent's target) and terminal agreement.
• Subjective Signal ($r_t$): A dense, per-turn score assigned by an LLM judge (Qwen3.5-Plus) evaluating the quality of the focal utterance based on anchoring, leverage, and consistency with the emotional stance.

2. Three-Stage Training Pipeline

EmoDistill utilizes these signals across three stages to train a 7B parameter SLM (Qwen2.5-7B-Instruct) with LoRA adapters:

• Stage 1: Emotion Selection (Implicit Q-Learning - IQL)

  • Goal: Learn which emotion to express given a dialogue state.
  • Mechanism: An IQL selector is trained on the offline dataset using the objective trajectory return $R(\tau)$ as the reward. It learns a policy $\pi_\phi(e|s)$ to select the optimal emotion action $e$ from a 28-label GoEmotions vocabulary.
  • Output: A selector that identifies high-value emotional sequences (e.g., transitioning from fear to anger) without updating the utterance generator.

• Stage 2: Expression Initialization (LoRA-SFT)

  • Goal: Learn how to express the selected emotion.
  • Mechanism: The SLM is fine-tuned via Supervised Fine-Tuning (SFT) using a hybrid filter. Demonstrations are selected based on a score $q_t^{hyb} = r_t + \frac{1}{2}R(\tau)$, ensuring the model imitates turns that are both locally well-formed (high judge score) and globally productive (high trajectory return).
  • Output: A LoRA adapter initialized to generate concrete, strategic utterances conditioned on the state and the selected emotion.

• Stage 3: Expression Refinement (Judge Policy Optimization - JPO)

  • Goal: Refine the expression policy using dense feedback.
  • Mechanism: JPO applies an offline PPO-clipped surrogate objective. It uses the per-turn judge advantage ($A_t$) to update the policy. Crucially, JPO employs an asymmetric advantage mechanism where negative-advantage samples are down-weighted by a factor $\kappa$. This allows the model to learn which specific emotional expressions improve the negotiation trajectory while preserving the "deal-closing" prior established during SFT.
  • Output: A refined policy that optimizes the specific wording of emotional expressions to maximize utility.

Key Contributions

1. EmoDistill Framework: The introduction of an offline pipeline that distills LLM-level emotional switching and expression strategies into a smaller 7B SLM, enabling high-performance negotiation without live API rollouts.
2. Decoupling Selection and Expression: Empirical evidence demonstrating that optimizing how an agent expresses a selected emotion (via LoRA-SFT and JPO) is distinct from and complementary to choosing the emotion (via IQL). The full pipeline outperforms baselines that only select emotions or only use fixed generators.
3. Judge Policy Optimization (JPO): A novel refinement method that leverages dense, turn-level LLM-judge advantages to improve emotional expression. Unlike sequence-level methods, JPO allows for fine-grained credit assignment, distinguishing between high-quality and low-quality expressions within the same negotiation trajectory.

Experimental Results

The framework was evaluated across four high-stakes negotiation domains: Credit Recovery (CRAD), Disaster Rescue, Hospital Surgery Scheduling, and Student Sleep Scheduling.

• Performance: The full EmoDistill policy (IQL+SFT+JPO) achieved the highest Utility (success rate weighted by deal quality) in three of the four domains, outperforming vanilla LLM/SLM baselines, random emotion prompting, and IQL-only baselines.
• Ablation Studies:
  • Component Analysis: Removing either the emotion selector (IQL) or the expression refinement (JPO) significantly degraded performance, confirming the necessity of both components.
  • Signal Analysis: SFT benefited most from hybrid quality filters (selecting good demonstrations), while JPO benefited most from dense turn-level judge advantages.
  • Emotion-Free Ablation: Removing the explicit emotion conditioning channel resulted in lower reliability and utility, suggesting that emotional strategy is not fully internalized without an explicit control interface.
• Transfer: The distilled skills showed partial transfer across domains and unseen counterparties (e.g., DeepSeek-V3, GPT-4o-mini). While success rates transferred well, value extraction (Utility) remained sensitive to domain-specific bargaining geometries.
• Risk Control: The parameter $\kappa$ in JPO allowed for tuning the trade-off between agreement success and per-deal value, with $\kappa=0.5$ offering an optimal balance in the CRAD domain.

Significance and Claims

The paper claims that emotion in negotiation should be treated as a controllable bargaining interface, not merely a generation style. By separating the selection of an emotional stance from the generation of the utterance, EmoDistill enables smaller models to learn effective, strategic bargaining behaviors that are robust against the "politeness bias" of standard LLMs.

The authors emphasize that their approach:

• Avoids Online Costs: By relying entirely on offline data from pre-judged LLM-vs-LLM interactions, it circumvents the prohibitive costs and instability of online RL in multi-agent settings.
• Provides Defensive Control: Modeling emotional influence as a learnable skill allows agents to resist manipulative emotional framing from counterparties, preserving user objectives in adversarial settings.
• Offers a Modular Skill: The framework treats emotional negotiation as a reusable skill that can be distilled, transferred, and refined, moving beyond static prompt engineering.

The authors maintain modesty regarding limitations, noting that the framework relies on fixed offline trajectories (risking distributional shift), requires an explicit emotion channel for optimal performance, and has not yet been validated in human-to-agent interactions. They position the work as a step toward more robust, strategic autonomous agents in high-stakes environments.