Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewarding

Technical Summary: Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewarding

Problem Statement
While Large Language Models (LLMs) excel in static, rule-based tasks, they struggle with dynamic social interactions, particularly in multi-agent negotiations requiring long-term goal coordination and rapid adaptation. Current optimization methods often apply uniform goal-based rewards to every utterance, which overlooks the specificity of objectives at each dialogue turn and fails to account for the rationale behind potential strategies. This approach risks training agents to identify "shortcut utterances" that maximize success metrics without genuinely learning human-like social inference or strategic planning. Furthermore, existing hierarchical methods often focus on strategy-level optimization while neglecting the stability and coherence of utterance-level execution.

Methodology
The authors propose the Think-Strategy-Response (TSR) framework, grounded in the Theory of Planned Behavior (TPB), to decompose social dialogue into two hierarchical stages:

1. High-Level Strategic Planning (Thinking & Strategy): The model first generates a "Thinking" phase (speculation, inference, analysis) and a "Strategy" phase (behavioral intention). This stage translates cognitive evaluation into an actionable plan, ensuring the response is guided by a specific, high-level strategy rather than a generic goal.
2. Low-Level Linguistic Execution (Response): Based on the generated strategy, the model produces the actual utterance. Notably, the "Thinking" content is excluded from the second stage's input to reduce computational overhead and prevent excessive reasoning from impairing performance.

To optimize this framework, the paper introduces Linearized Hierarchical Reinforcement Learning with Variance-Gated Rewards (LHRL-VGR). This algorithm employs a novel reward routing mechanism:

• Composite Reward for Strategy: The strategy generator is optimized using a composite reward comprising Predicted Outcome Rewards (estimating progress toward the goal) and Content Completeness Rewards (evaluating alignment with behavioral guidance and utterance-level objectives).
• Variance-Gated Reward for Response: The response generator utilizes a dynamic reward router. It calculates the variance of goal achievement scores across multiple response samples.
  • If the variance exceeds a threshold ($\theta$), indicating uncertainty in goal completion, the system prioritizes the Goal-completion Reward ($r_{goal}$).
  • If the variance is low (stable), the system prioritizes the Strategy-followed Reward ($r_{follow}$), ensuring adherence to the planned strategy.
• Two-Stage Group Relative Policy Optimization (GRPO): The training process jointly optimizes both stages using a Group Relative Policy Optimization algorithm. It calculates advantages for strategy content and transferred goal completion rewards, balancing them via a hyperparameter $\alpha$.

Key Contributions

1. TSR Framework: A linearized hierarchical paradigm that explicitly separates strategic planning from linguistic execution, enabling more rational and interpretable dialogue generation.
2. LHRL-VGR Algorithm: A novel training algorithm featuring a variance-gated reward router that dynamically assigns either goal-completion or strategy-adherence rewards to the response generator, while jointly training both policy levels through a composite strategic reward.
3. State-of-the-Art Performance: The method achieves superior results on the SOTOPIA benchmark, demonstrating that a Qwen2.5-7B agent fine-tuned with LHRL-VGR outperforms the GPT-4o baseline.

Experimental Results
Experiments were conducted on the SOTOPIA and SOTOPIA-Hard benchmarks, evaluating agents in multi-turn dialogues against GPT-4o as a partner.

• Performance: The LHRL-VGR fine-tuned Qwen2.5-7B agent achieved a 7.32% relative improvement in goal completion success over the GPT-4o baseline on the SOTOPIA benchmark.
• Ablation Studies: Removing the strategy completeness reward or the transferred goal completion reward led to significant performance drops, confirming the necessity of joint optimization.
• Training Paradigm: Coordinated joint training of both stages outperformed sequential training (Stage 1 then Stage 2, or vice versa) and single-stage RL approaches, particularly on human-evaluated metrics for strategy and response quality.
• Reward Routing: The variance-gated routing mechanism significantly outperformed alternative routing strategies based on pre-defined dialogue modes or strategy-predicted modes.

Significance and Claims
The paper claims that the proposed approach addresses two core challenges in enhancing social intelligence for LLMs:

1. Genuine Social Inference: By explicitly generating and optimizing strategies, the method prevents models from learning shortcut utterances, instead fostering human-like capabilities such as intent reasoning and strategic planning.
2. Dynamic Reward Adaptation: The variance-gated mechanism overcomes the limitation of applying uniform goal-based rewards, allowing the model to adapt to the distinct objectives of different utterances by balancing goal completion with strategy adherence.

The authors conclude that their method offers a promising path toward more human-like and intelligent LLM-based agents in complex social negotiation tasks, achieving state-of-the-art performance while maintaining interpretability through the structured TSR framework.