Adaptive Social Learning via Mode Policy Optimization for Language Agents

This paper proposes the Adaptive Social Learning (ASL) framework, featuring the Adaptive Mode Policy Optimization (AMPO) algorithm, to enable language agents to dynamically switch between intuitive and deliberative reasoning modes based on context, thereby achieving superior task performance and token efficiency compared to existing methods like GPT-4o and GRPO.

The Gist

Here is an explanation of the paper "Adaptive Social Learning via Mode Policy Optimization for Language Agents" using simple language and creative analogies.

The Big Problem: The "Over-Thinker" vs. The "Impulsive" Robot

Imagine you are at a party.

• Scenario A: Someone asks you, "What's the weather like?" You instantly say, "It's sunny." You didn't need to think about this; it's a quick, intuitive reaction.
• Scenario B: Someone asks, "Can I borrow your car for a week?" You don't answer immediately. You pause. You think about your schedule, your insurance, your relationship with them, and what might happen if they crash it. You deliberate deeply before answering.

Current AI language models (LLMs) are terrible at knowing when to do which.

• Some models are like impulsive robots: They answer everything instantly without thinking, often missing the point in complex social situations.
• Other models (the new "Reasoning Models" like o1 or DeepSeek) are like obsessive over-thinkers. Even if you ask them "What's 2+2?", they write a 500-word essay analyzing the history of mathematics before giving the answer. This wastes time, costs money (in computing power), and feels unnatural in a conversation.

The Paper's Goal: Create an AI agent that acts like a wise human: knowing exactly when to snap back with a quick answer and when to pause and strategize deeply.

---

The Solution: The "Cognitive Toolkit" (ASL Framework)

The authors propose a framework called ASL (Adaptive Social Learning). Think of this as giving the AI a Swiss Army Knife with four distinct tools, and a smart handle that knows which tool to pick up for the job.

1. The Four "Thinking Modes"

Based on how human brains work, the AI is trained to switch between four specific "modes":

• Mode 1: The Reflex (Intuitive Response)
  • Analogy: Like pulling your hand away from a hot stove.
  • Use: Simple greetings or obvious facts. No thinking required.
• Mode 2: The Chat (Intentional Analysis)
  • Analogy: Like a casual coffee chat. You listen to what the other person said, check your tone, and reply.
  • Use: Normal conversation where you just need to be polite and relevant.
• Mode 3: The Strategist (Strategic Adaptation)
  • Analogy: Like a chess player looking 2 moves ahead. You look at the history of the conversation, your long-term goal, and the current situation to pick a strategy.
  • Use: Negotiations or when you need to be careful not to offend someone.
• Mode 4: The Simulator (Prospective Deduction)
  • Analogy: Like a movie director running a "what-if" scene in their head. You imagine three different ways to say something, predict how the other person will react to each, and then pick the best one.
  • Use: High-stakes situations, like resolving a fierce argument or a complex business deal.

2. The "Smart Switch" (AMPO Algorithm)

The real magic isn't just having these modes; it's the Adaptive Mode Policy Optimization (AMPO) algorithm. This is the "brain" that decides which mode to use.

• The Old Way (GRPO): Imagine a student who always studies for 5 hours for every test, whether it's a pop quiz or a final exam. It's inefficient and exhausting.
• The New Way (AMPO): Imagine a student who looks at the test. If it's a pop quiz, they spend 5 minutes. If it's a final exam, they spend 5 hours. They learn to adapt their effort to the difficulty of the situation.

AMPO teaches the AI to look at the social context and ask: "Is this a simple question, or is this a life-or-death negotiation?" Then, it switches the mode accordingly.

---

How They Taught the AI (The Training Process)

The researchers didn't just tell the AI to "be smart." They used a two-step training process:

1. Imitation (Behavioral Cloning): First, they showed the AI thousands of examples of humans (or expert AIs) solving social problems. They taught the AI: "When you see this situation, use Mode 3. When you see that, use Mode 1." This is like a student memorizing the answer key.
2. Reinforcement Learning (The "Reward" Game): Then, they let the AI play social games (like negotiating or collaborating).
   • If the AI used a deep, complex mode for a simple question, they gave it a "penalty" (like saying, "You wasted time!").
   • If the AI used a quick mode for a complex problem and failed, they gave it a penalty ("You didn't think enough!").
   • If it picked the right mode and got a good result, it got a "reward."

Over time, the AI learned the perfect balance: Do the minimum amount of thinking necessary to get the best result.

---

The Results: Smarter and Faster

The paper tested this new AI against the best existing models (including GPT-4o and other "Reasoning Models").

• Better Performance: The new AI won more social challenges (like negotiations) than the experts. It achieved 15.6% better results than GPT-4o.
• More Efficient: Because it stopped over-thinking simple tasks, it used 32.8% fewer words (tokens) to get the job done compared to other reasoning models.
• Human-Like: In human evaluations, people felt the AI's conversations were more natural, strategic, and effective.

The Takeaway

This paper solves the "Goldilocks" problem of AI reasoning.

• Old models were either too cold (no thinking) or too hot (thinking too much).
• This new framework creates an AI that is just right. It knows when to be a quick-witted friend and when to be a deep-thinking strategist, making it a much better companion for real-world social interactions.

Technical Summary

Here is a detailed technical summary of the paper "Adaptive Social Learning via Mode Policy Optimization for Language Agents" (ICLR 2026).

1. Problem Statement

Current Large Language Model (LLM) agents struggle with social intelligence, particularly in dynamic, open-ended interactions involving conflicting interests (e.g., negotiation, collaboration). Existing approaches suffer from two main limitations:

• Lack of Explicit Reasoning: Many methods rely on "fast-reasoning" (end-to-end training or external planners) that lack deep strategic thinking, leading to responses that miss subtle social cues or long-term goals.
• Inflexible Reasoning Depth: State-of-the-art Large Reasoning Models (LRMs) like OpenAI-o1 or DeepSeek-R1 employ Long Chain-of-Thought (Long-CoT) uniformly. They exhaustively reason regardless of task complexity. In social contexts, this leads to "overthinking," excessive token usage, and inflexible behaviors that degrade performance in simple interactions or fail to adapt to evolving social dynamics.

The core challenge is enabling agents to dynamically adjust their reasoning depth based on the social context, switching between intuitive responses and deep deliberation as needed.

2. Methodology: Adaptive Social Learning (ASL) Framework

The authors propose the ASL framework, which consists of three main stages:

A. Hierarchical Reasoning Mode Design

Inspired by Hierarchical Cognitive Control Theory (HCCT), the authors define four distinct reasoning modes to structure the agent's cognitive process:

1. Mode 1 (Intuitive Response): Immediate reaction based on learned associations; no explicit reasoning steps (System 1 thinking).
2. Mode 2 (Intentional Analysis): Shallow reasoning focusing on current intent, style, and a preliminary response.
3. Mode 3 (Strategic Adaptation): Moderate reasoning incorporating dialogue history, goal clarification, situation assessment, and a single strategy proposal.
4. Mode 4 (Prospective Deduction): Deep deliberation involving the generation of multiple strategies, simulating their outcomes (deduction), and integrating results for an optimal decision.

B. Mode Behavioral Cloning (BC)

To teach the model to adhere to these specific modes, the authors first perform Behavioral Cloning. An expert LLM generates high-quality training data where responses strictly follow the predefined structure of the four modes (including specific tags for actions like History, Goal, Strategy, Deduction, etc.). This serves as a cold-start foundation for the model.

C. Adaptive Mode Policy Optimization (AMPO)

The core innovation is the AMPO algorithm, a Reinforcement Learning (RL) method designed to learn context-aware mode switching. Unlike standard PPO or GRPO (Group Relative Policy Optimization), AMPO introduces a dual-level advantage estimation:

1. Reward Shaping:

   • Answer Reward: Measures goal completion (evaluated by an LLM judge).
   • Format Reward: Penalizes deviations from the required reasoning mode structure.
   • Length Reward: Penalizes excessive token usage to encourage conciseness when deep reasoning is unnecessary.

2. Dual-Level Advantage Estimation:

   • Mode-Level Advantage ($A_M$): Evaluates the performance and efficiency of the chosen reasoning mode across a batch of samples. If a mode yields high rewards with low token counts, it is favored. This guides the agent to select the right level of thinking for the scenario.
   • Sample-Level Advantage ($A_S$): Evaluates the quality of the specific output generated within that chosen mode, ensuring the content is better than the group average.

The objective function combines these advantages to optimize the policy, encouraging the agent to switch modes dynamically (e.g., using Mode 1 for simple greetings and Mode 4 for complex negotiation stalemates).

3. Key Contributions

1. First Adaptive Social Learning Framework: ASL is the first framework to explicitly enable language agents to adaptively switch reasoning depths in dynamic social interactions, bridging the gap between fast social responses and deep strategic reasoning.
2. AMPO Algorithm: A novel RL algorithm that integrates mode-level and sample-level advantage estimation. It solves the "mode-blindness" of existing methods (like GRPO) by explicitly rewarding the selection of efficient reasoning modes.
3. Token Efficiency & Performance: The framework achieves superior performance while significantly reducing token consumption compared to uniform Long-CoT approaches.

4. Experimental Results

The framework was evaluated on SOTOPIA and SOTOPIA-Hard benchmarks, which test social intelligence in role-playing scenarios.

• Performance Gains:
  • ASL (with AMPO) achieved SOTA performance, outperforming GPT-4o by 15.6% on the SOTOPIA-Hard benchmark (using a Llama3.1-8B backbone).
  • It significantly outperformed other baselines including proprietary models (Claude-3.5, DeepSeek-V3) and other reasoning models (OpenAI-o1, DeepSeek-R1).
• Efficiency vs. GRPO:
  • Compared to GRPO (the standard for training LRMs), AMPO achieved a 7.0% higher performance score while using 32.8% fewer tokens on average.
  • Training dynamics show that GRPO tends to converge to a single, overly complex mode (Mode 4), whereas AMPO successfully learns to distribute usage across all four modes based on context.
• Adaptive Behavior Analysis:
  • Experiments confirmed that AMPO uses complex modes (M3/M4) primarily in early turns or difficult contexts (e.g., when goals are unmet) and shifts to simpler modes (M1/M2) as the conversation progresses or goals are achieved.
• Human Evaluation:
  • Human annotators confirmed that AMPO agents produce more effective, relationship-building, and goal-oriented dialogues compared to GRPO and behavioral cloning baselines, with no evidence of "reward hacking."

5. Significance

This paper addresses a critical bottleneck in AI social intelligence: the inability of current models to know when to think deeply and when to act intuitively.

• Theoretical Impact: It successfully translates cognitive control theories (human decision-making) into a practical RL framework for LLMs.
• Practical Impact: By reducing token usage by ~30% while improving task success, ASL offers a cost-effective and scalable solution for deploying social agents in real-world applications (e.g., customer service, negotiation bots, companionship).
• Future Direction: It establishes a new paradigm for "adaptive reasoning," moving beyond static Long-CoT towards dynamic, context-sensitive cognitive architectures.

The code and data are publicly available, facilitating further research into adaptive social agents.