SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution

1. Problem Statement

The paper addresses the challenge of training language agents to exhibit social intelligence—the ability to navigate complex, multi-turn interpersonal interactions to achieve social goals. While Reinforcement Learning (RL) is a promising approach, it suffers from a critical credit assignment problem: determining how individual utterances contribute to the final outcome of a dialogue.

Existing methods (e.g., Sotopia-RL) rely on Large Language Models (LLMs) to heuristically distribute episode-level rewards to individual turns. These approaches have two fundamental flaws:

1. Lack of Theoretical Grounding: The credit distribution is heuristic, lacking guarantees of fairness or accuracy.
2. Retrospective Bias: They evaluate utterances based on their contribution to the observed outcome. This fails to capture strategic potential, where an utterance might seem minor in the immediate context but unlocks favorable future trajectories (prospective value).

2. Methodology: The SAVOIR Framework

The authors propose SAVOIR (ShApley Value fOr SocIal RL), a framework grounded in cooperative game theory to solve the credit assignment problem. It integrates two complementary principles:

A. Expected Utility for Prospective Valuation

Instead of asking "What did this utterance contribute to the final result?", SAVOIR asks, "What is the expected value of future interactions given this utterance?"

• Mechanism: For a subset of utterances (a coalition $S$), the framework reconstructs the dialogue history and performs Monte Carlo rollouts (simulating future dialogues with a partner policy).
• Value Function ($v(S)$): The value is the expected utility of these simulated future trajectories, calculated using a multi-dimensional scoring system (Goal, Believability, Relationship, etc.) from the SOTOPIA benchmark.
• Benefit: This captures the strategic potential of an utterance, rewarding actions that set up favorable conditions even if the immediate reward is low.

B. Shapley Values for Fair Credit Attribution

Once the value of coalitions is established, the framework must distribute this total value fairly among individual utterances.

• Mechanism: SAVOIR employs Shapley values, the unique solution in cooperative game theory that satisfies axioms of efficiency, symmetry, and marginality.
• Computation: The Shapley value $\phi_i$ for an utterance $i$ is the weighted average of its marginal contribution across all possible orderings of utterances.
• Optimization: Since calculating exact Shapley values is computationally prohibitive ($2^n$ subsets), the authors use KernelSHAP. This reformulates the problem as a weighted linear regression, prioritizing coalitions of extreme sizes (very small or very large) to efficiently approximate the values with a limited number of rollouts.

C. Training Pipeline

1. Data Collection: Agents engage in self-play to generate social interaction episodes.
2. Reward Modeling: The SAVOIR algorithm computes fine-grained, utterance-level rewards ($\hat{\phi}_i$) for these episodes.
3. Policy Training: A reward model ($R_\theta$) is trained to predict these SAVOIR scores. The agent policy is then optimized via Supervised Fine-Tuning (SFT) followed by Online RL (GRPO) using the learned reward model.

3. Key Contributions

1. Theoretical Framework: Introduction of SAVOIR, the first framework to combine Expected Utility (for prospective strategic valuation) and Shapley Values (for axiomatic, fair credit distribution) in social RL.
2. State-of-the-Art Performance: Demonstrated that a 7B parameter model (Qwen2.5-7B) trained with SAVOIR matches or exceeds proprietary models (GPT-4o, Claude-3.5-Sonnet) on the SOTOPIA benchmark.
3. Insight on Reasoning vs. Social Intelligence: Revealed that Large Reasoning Models (e.g., OpenAI-o1, DeepSeek-R1) consistently underperform on social tasks. This suggests that social intelligence relies on intuitive, strategic capabilities distinct from the deliberative, analytical reasoning these models excel at.
4. Human Validation: Expert human evaluation confirmed that SAVOIR produces more strategic responses and that its reward model aligns better with human intuition regarding credit fairness and future potential.

4. Experimental Results

• Benchmark: Evaluated on SOTOPIA (both "All" and "Hard" splits) with two settings: Self-Play and GPT-4o-as-Partner.
• Performance:
  • On SOTOPIA-Hard (GPT-4o partner), SAVOIR achieved a Goal score of 7.18, improving over the strongest baseline (Sotopia-RL) by 7.5%.
  • In Self-Play, SAVOIR (7.93) outperformed DSI (7.31) and Sotopia-RL (7.81).
  • The 7B SAVOIR model outperformed GPT-4o (8.19 vs 8.43 in Self-Play All) and Claude-3.5-Sonnet.
• Ablation Studies:
  • Removing Expected Utility (using only final outcomes) reduced performance.
  • Removing Shapley Values (using heuristic attribution) also reduced performance.
  • The combination yielded the best results, proving the components address orthogonal limitations (value estimation vs. distribution).
• Scaling: Increasing the reward model training data from 2K to 7.5K episodes led to a 15.2% increase in Goal scores, indicating the method scales well with data.

5. Significance and Implications

• Paradigm Shift: SAVOIR moves social RL from retrospective, heuristic reward assignment to prospective, principled valuation. This allows agents to learn "social savoir-faire"—the art of acting appropriately to enable future success.
• Nature of Social Intelligence: The underperformance of reasoning-heavy models suggests that social interaction requires a qualitatively different skillset (intuition, rapport building, strategic positioning) that is not merely a function of increased computational depth or chain-of-thought reasoning.
• Future Directions: The work bridges Game Theory and AI, offering a robust mathematical foundation for credit assignment in multi-agent systems. It highlights the need for curriculum learning with diverse partners and cross-cultural evaluation to further generalize social intelligence.

In conclusion, SAVOIR demonstrates that by mathematically formalizing the "value" of social moves through game theory, smaller, specialized models can outperform massive, general-purpose reasoning models in complex social environments.