💬 NLP

ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation

This paper introduces ProMediate, the first socio-cognitive framework and simulation testbed for systematically evaluating proactive AI mediator agents in complex multi-party negotiations, demonstrating that socially intelligent agents significantly outperform generic baselines in both consensus-building effectiveness and intervention speed.

Original authors: Ziyi Liu, Bahar Sarrafzadeh, Pei Zhou, Longqi Yang, Jieyu Zhao, Ashish Sharma

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyi Liu, Bahar Sarrafzadeh, Pei Zhou, Longqi Yang, Jieyu Zhao, Ashish Sharma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🌟 The Big Idea: The "Group Chat Therapist"

Imagine you are in a group chat with five friends trying to decide where to go for dinner. One friend wants pizza, another wants sushi, a third is on a diet, and a fourth is just hungry and angry. The conversation gets messy. People talk over each other, get frustrated, and eventually, the group gives up and orders takeout they all hate.

Now, imagine a super-smart, invisible friend (an AI) watching this chat. This friend doesn't just wait for someone to ask for help. Instead, they are proactive. They notice the tension rising, they see that the "sushi" friend is getting annoyed at the "pizza" friend, and they jump in before the fight gets ugly to suggest a compromise: "How about we get sushi and pizza, and split the bill?"

This paper introduces PROMEDIATE, a new testing ground (a "gym") to train and test these kinds of AI "group chat therapists" (called Mediator Agents).

🏗️ What is PROMEDIATE?

Think of PROMEDIATE as a virtual negotiation simulator. It's like a flight simulator for pilots, but instead of crashing planes, the AI is trying to stop human arguments from crashing.

It has two main parts:

1. The Simulation Environment (The "Playground")

The Scenario: The AI is dropped into realistic, messy situations. For example, a pharmaceutical company and a hospital are arguing over the price of a new drug. They have to agree on discounts, how much of the drug to buy, and how long the contract lasts.
The Characters: The "humans" in the simulation are actually other AIs programmed with specific personalities. Some are Accommodating (easy to please), some are Avoiding (they ignore the problem), and some are Competing (they are stubborn and only care about winning).
The Plug-and-Play Mediator: The AI mediator is a "plug-in." You can swap it out. You can test a "Generic" AI (who just says "Let's agree") versus a "Socially Intelligent" AI (who understands emotions and psychology).

2. The Socio-Cognitive Scorecard (The "Report Card")

How do we know if the AI mediator is good? We don't just look at whether they got a deal. We look at how they did it. The paper uses a new set of metrics:

Consensus Change: Did the group move from "I hate this" to "Okay, this works"?
Intervention Speed: Did the AI jump in when things started to go wrong, or did it wait too long?
Mediator Intelligence: Did the AI fix the real problem?
- Example: If two people are arguing because they misunderstood each other (Perceptual), did the AI clarify that? If they were yelling because they were angry (Emotional), did the AI calm them down?

🧪 The Experiment: What Happened?

The researchers tested three types of mediators in different "difficulty levels":

No Agent: Just the humans arguing. (Usually ends in a deadlock).
Generic Mediator: An AI that tries to help but doesn't really understand human psychology. It's like a referee who just blows a whistle when people yell.
Socially Intelligent Mediator: An AI trained to understand why people are fighting. It knows when to listen, when to validate feelings, and when to suggest a clever compromise.

The Results:

In Easy Scenarios: When the humans were already nice and willing to compromise, the fancy AI didn't add much value. Sometimes, it even got in the way!
In Hard Scenarios: When the humans were stubborn, angry, and competitive (the "Hard" setting), the Socially Intelligent Mediator was a game-changer.
- It helped the group reach 3.6% more agreement than the generic AI.
- It was 77% faster at spotting the problem and jumping in.
- It didn't just force a deal; it actually helped the group understand each other better.

💡 Why Does This Matter? (The "So What?")

Most AI today is like a personal assistant. You ask it to "write an email," and it does.

But the real world is full of group projects where people have different goals, egos, and emotions. We need AI that acts like a skilled team captain or a diplomat.

The Problem: Current AI is often too passive. It waits to be told what to do.
The Solution: PROMEDIATE proves that AI can be proactive. It can sense the "vibe" of a room, predict a fight before it happens, and steer the conversation toward a solution.

🚀 The Takeaway

PROMEDIATE is a new tool that helps us build AI that is not just smart, but socially smart. It teaches AI how to be the perfect mediator in a room full of conflicting opinions, ensuring that groups don't just survive their arguments, but actually find a solution that works for everyone.

It's the difference between a robot that says, "Error: Disagreement detected," and a robot that says, "Hey, I see we're stuck. Let's try looking at this from a different angle so we can all win."

1. Problem Statement

While Large Language Models (LLMs) are increasingly used to assist individual users, there is a critical gap in developing and evaluating agents capable of proactively managing complex, multi-party collaborations. Existing benchmarks often focus on:

Reactive agents: Systems that only respond when explicitly prompted.
Bilateral interactions: One-on-one negotiations that lack the socio-cognitive complexity of group dynamics.
Simplified settings: Game-based scenarios that fail to capture real-world negotiation nuances (e.g., multiple topics, conflicting interests, emotional dynamics).

The paper argues that effective multi-party negotiation requires socio-cognitive intelligence: the ability to track multiple perspectives, anticipate divergent attitudes, and proactively steer discussions toward shared outcomes. There is currently no systematic framework to evaluate how well AI agents can intervene in real-time to resolve deadlocks in multi-party settings.

2. Methodology: The PROMEDIATE Framework

The authors introduce PROMEDIATE, the first testbed designed to evaluate proactive AI mediator agents. It consists of two core components:

A. Simulation Environment (The Testbed)

Realistic Scenarios: Based on Harvard Law School negotiation training materials, the framework uses six complex scenarios (e.g., healthcare contracts, environmental policy) involving multiple parties, multiple topics, and multiple options per topic.
Plug-and-Play Architecture: The system simulates human participants using LLMs (specifically Claude-Sonnet-4, chosen for its natural conversational behavior) with structured preference profiles. A proactive AI mediator is inserted as a modular agent that observes the conversation and decides when and how to intervene.
Conflict Modes: To vary difficulty, the framework employs three conflict modes based on established theory (Cai & Fink, 2002):
- Accommodating: Participants yield to others (Easy).
- General/Avoiding: Participants sidestep conflict (Medium).
- Competing: Participants hold firm to their own interests (Hard).
Human Simulation: Uses the InnerThought framework, allowing simulated humans to generate internal thoughts and intrinsic motivation scores to determine speaking turns, ensuring dynamic and realistic conversation flow.

B. Socio-Cognitive Evaluation Framework

Instead of relying solely on final outcomes, PROMEDIATE introduces a suite of metrics grounded in socio-cognitive theory to measure the process and intelligence of mediation:

Consensus Tracking: A soft, time-varying measure of agreement.
- Attitude Extraction: An LLM infers participant stances on specific topics from utterances without relying on fixed option sets.
- Agreement Scoring: Computes pairwise agreement scores (0–1) across five dimensions: Shared Goals, Common Understanding, Agreement on Terms, Tone/Willigness, and Shared Decision Making.
Performance Metrics:
- Consensus Change (CC): Improvement in consensus from start to end (windowed average).
- Topic-Level Efficiency (TLE): Rate of agreement change per topic mention.
- Response Latency (RL): Time taken to intervene after a consensus drop.
- Mediator Effectiveness (ME): The slope of consensus improvement immediately following an intervention.
- Mediator Intelligence (MI): A score (1–5) evaluating if the intervention addresses four socio-cognitive dimensions: Perceptual Differences, Emotional Dynamics, Cognitive Challenges, and Communication Breakdowns.

3. Key Contributions

Extensible Testbed: A unified environment for simulating realistic multi-party negotiations with configurable conflict modes and plug-and-play mediator agents.
Socio-Cognitive Integration: The first framework to ground both agent design and evaluation metrics in mediation theory and socio-cognitive frameworks, moving beyond simple task completion.
Comprehensive Metrics: A multi-dimensional evaluation suite that decouples "process quality" (Intelligence) from "immediate outcome" (Effectiveness), revealing that high-quality interventions do not always yield immediate consensus gains.
Agent Comparison: A rigorous comparison between a Generic Mediator (rule-based, reactive) and a Socially Intelligent Mediator (theory-driven, proactive).

4. Experimental Results

The authors evaluated three settings (NoAgent, Generic Mediator, Socially Intelligent Mediator) across three difficulty levels (Easy, Medium, Hard) using different LLM backbones (GPT-4.1, o4-mini, Claude-Sonnet-4).

Performance in Hard Settings: In the PROMEDIATE-Hard (Competing) setting, the Socially Intelligent Mediator significantly outperformed the Generic baseline:
- Consensus Change: Increased by 3.6 percentage points (10.65% vs. 7.01%).
- Response Latency: Was 77% faster (3.71s vs. 15.98s), demonstrating the ability to detect breakdowns earlier.
Context Dependence: In Easy settings (Accommodating), proactive intervention offered little added value and sometimes disrupted organic consensus. This highlights that effective mediation requires adaptive strategies calibrated to the conflict intensity.
Model Performance: The o4-mini model achieved the highest consensus change (9.34%) despite having the highest latency, suggesting that deliberate reasoning leads to higher-quality interventions.
Construct Validity: Factor analysis revealed two distinct latent factors:
1. Consensus & Topic Efficiency (Outcome).
2. Intervention Dynamics/Tempo (Process).
- Crucially, Mediator Intelligence and Mediator Effectiveness showed no significant correlation (Spearman $\rho \approx 0.01$ ). High-quality interventions often caused temporary drops in consensus (constructive friction) as latent disagreements were surfaced, aligning with negotiation theory (e.g., the "storming" phase).

5. Significance and Conclusion

PROMEDIATE establishes a rigorous, theory-grounded standard for developing proactive, socially intelligent AI agents. The paper demonstrates that:

Proactivity is context-dependent: Agents must adapt their intervention frequency and timing based on the group's conflict mode.
Process $\neq$ Outcome: A mediator can be highly intelligent (addressing emotional/cognitive root causes) without immediately increasing consensus scores, as resolving deep conflicts often requires temporary discord.
Future Direction: The framework provides a necessary foundation for building AI that can responsibly facilitate real-world group decisions, moving beyond simple task assistance to complex social orchestration.

The work emphasizes that no single metric can capture a mediator's full capability, necessitating a multi-faceted evaluation approach that considers both the efficiency of the outcome and the socio-cognitive quality of the intervention.