💬 NLP

Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue

The paper introduces MASP, a Mentalized Adversarial Self-Play framework that models proactive dialogue as a partially observable game against a rationally resisting user with a BDI-based mental state, thereby training agents to achieve robust task success and efficiency even against difficult, non-compliant users.

Original authors: Zanwu Liu, Dan Zhang, Xiaodi Wang, Yongbo Mao, Xiaoyang Guo, Yonglin Wu, Peiyue Jiang, Yangyang Wei, Zecheng Lin

Published 2026-10-16✓ Author reviewed ⓘ

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Zanwu Liu, Dan Zhang, Xiaodi Wang, Yongbo Mao, Xiaoyang Guo, Yonglin Wu, Peiyue Jiang, Yangyang Wei, Zecheng Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a distinct difference between a chatbot that simply replies to what you say and one that actively guides a conversation toward a specific goal. This second type, known as proactive dialogue, is the engine behind digital assistants that try to persuade you to donate to charity, help you calm down when you are distressed, or negotiate a price for a used item. For these systems to work, they must do more than just generate fluent sentences; they must anticipate how their words will change your mind, your feelings, and your intentions over the course of a long conversation. The challenge has always been that the most advanced AI models are often trained in a vacuum, practicing their skills against computer programs that are too eager to please. These programs, or simulators, act like overly compliant students who agree to everything, allowing the AI to score high marks without ever learning how to handle a real person who might be skeptical, resistant, or simply changing their mind.

A team of researchers from Dongchezu Technology and Beihang University has introduced a new training method designed to fix this flaw. They call their approach MASP, which stands for Mentalized Adversarial Self-Play. Instead of letting the AI practice on a soft, easy target, the researchers set up a training ground where the AI plays against a digital opponent that is designed to be rational but difficult. In this setup, the AI must learn to read the hidden mental state of the user—what they believe, what they want, and what they intend to do—and adjust its strategy accordingly. The system is rewarded not just for winning, but for understanding the user well enough to make progress even when the user is pushing back. This method forces the AI to develop a robust ability to navigate complex human interactions, rather than just memorizing how to win against a compliant script.

The core idea behind this work is that an AI cannot learn to guide a conversation if it never encounters genuine resistance. In previous methods, the AI would train against a fixed user model that was either static or too easy to manipulate. The researchers found that this led to a "soft user" problem, where the AI appeared successful in tests but failed when faced with a real person who had their own evolving thoughts. To solve this, the researchers created a training loop where the AI and a user simulator play against each other in a continuous game. The user simulator is not just a random blocker; it is trained to resist the AI's suggestions in a rational way. If the AI offers a good reason, the user might agree. If the AI offers a weak argument, the user will push back. This dynamic forces the AI to learn how to build a convincing case step by step, rather than relying on lucky breaks or overly simple tactics.

A crucial part of this system is how it tracks the user's internal state. The researchers use a framework based on beliefs, desires, and intentions. Imagine a hidden map that the AI tries to read, which shows what the user currently thinks, what they hope to achieve, and what they plan to do next. The AI does not see this map directly; it only sees the words the user says and must infer the map from the conversation. To make sure the AI learns to infer correctly, the training setup uses a "teacher" and a "student" approach. A frozen teacher model, which has been pre-trained on a large amount of data, knows the true hidden state of the user. The AI, acting as the student, only sees an estimate of that state. The AI is then rewarded for making progress toward the goal while also being penalized if its estimate of the user's mind is wrong. This ensures the AI learns to understand the person it is talking to, not just the words they are saying.

The researchers tested this method across several different types of conversations, including persuading people to donate, providing emotional support, showing empathy, and negotiating prices. They compared their new method against two other common approaches: one where the AI is simply trained on human conversations without any special strategy, and another where the AI uses a planner but still trains against an easy user. The results showed that the new method consistently outperformed the others. In the persuasion tests, the AI achieved a success rate of nearly 86 percent against easy users and 79 percent against difficult ones, while the other methods struggled significantly more with the difficult users. The new method also reduced the number of turns needed to reach a goal, meaning the conversations were more efficient. Perhaps most importantly, the AI trained with this method was better at maintaining a rational conversation, avoiding situations where it would force a win by confusing or annoying the user.

The study also looked at how the AI performed when the user was particularly stubborn. In these hard scenarios, the new method maintained a high success rate, whereas the other methods saw their performance drop sharply. This suggests that the AI learned a more flexible and resilient strategy. It did not just learn to win against one specific type of user; it learned to adapt to a wide range of human behaviors. The researchers noted that the AI's ability to succeed was not just about being faster or more aggressive, but about being able to read the room and adjust its approach based on the user's changing mental state. When the user was resistant, the AI would pause and gather more information. When the user showed signs of openness, the AI would make its move.

One of the key findings was that the AI's success was tied to its ability to understand the user's rationality. The researchers introduced a new way of measuring success that only counted a win if the user agreed for a logical reason, rather than just giving in to pressure. Under this stricter measure, the new method still performed better than the alternatives. This indicates that the AI was not just tricking the user into agreeing; it was actually guiding them to a conclusion that made sense to them. The training process, which involved thousands of simulated conversations where the AI and the user played against each other, allowed the system to refine its strategy over time. The AI learned that the best way to win was to build trust and provide clear, honest reasons for its suggestions.

The researchers also conducted a detailed analysis of where the system failed. In the cases where the AI did not succeed, it was often because the user had a strong financial constraint or a deep-seated lack of trust that the AI could not overcome in the limited number of turns. These failures were not due to the AI being confused or aggressive, but rather because the user's internal state was too resistant to change within the given timeframe. This analysis helped the researchers understand the limits of the system and confirmed that the AI was behaving rationally even in failure. It did not persist in asking for a donation after the user had clearly said no; instead, it recognized the impasse and stopped.

This work represents a significant step forward in making AI agents that can interact with humans in a more natural and effective way. By training the AI against a rational but difficult opponent, the researchers have created a system that is better prepared for the complexities of real human conversation. The method does not rely on magic or complex tricks; it relies on the simple but powerful idea that to learn how to persuade, you must first learn how to listen and understand. The results suggest that this approach could be applied to many other areas where AI needs to guide human behavior, from mental health support to customer service. The study concludes that treating the user as an active, rational participant in the training process is essential for building AI that can truly succeed in the real world.

The researchers are careful to note that their system is still a simulation and has not yet been tested on real people in a live setting. The success rates and behaviors described are based on interactions between computer programs. However, the consistency of the results across different types of conversations and the robustness of the system against difficult users provide strong evidence that the method works. The team plans to continue refining the system and eventually test it with real users to see how it performs outside the controlled environment of the simulation. For now, the study offers a compelling new direction for how we teach machines to talk to us, moving away from simple pattern matching toward a deeper, more strategic understanding of human thought.

Technical Summary: Mentalized Adversarial Self-Play for Proactive Dialogue (MASP)

Problem Statement

Proactive dialogue agents must guide multi-turn interactions toward specific task goals (e.g., persuasion, negotiation, emotional support) while adapting to users whose intentions and attitudes evolve. Existing approaches typically improve system-side action selection through search, learned planners, or user-aware planning. However, these methods often suffer from a critical training asymmetry: the system is optimized as the sole strategic learner, while the user is modeled as a fixed, externally specified, or overly compliant simulator.

This asymmetry leads to "soft user bias," where policies achieve high simulated success rates by learning brittle behaviors that fail when confronted with rational, resistant, or strategic users. Current efforts to improve user simulation (e.g., dedicated User LMs, iterative adversarial refinement) primarily strengthen the evaluation environment but do not directly address how a proactive policy should be trained to survive against stronger, partially observable, and strategically resisting users. Furthermore, standard evaluation metrics often fail to distinguish between genuine task success and success achieved through simulator sycophancy or unjustified concessions.

Methodology: MASP Framework

The authors propose MASP (Mentalized Adversarial Self-Play), a training paradigm that formulates proactive dialogue as a partially observable game between a system agent and a rationally resisting user agent over a shared latent mental state space.

1. Problem Formulation

The interaction is modeled as a game where:

User Policy ( $\pi_\psi$ ): Conditions on a hidden latent mental state $z_t$ representing the user's evolving internal state.
System Policy ( $\pi_\theta$ ): Observes only an estimate $\hat{z}_t$ inferred from the dialogue history.
Asymmetry: The system does not have direct access to the user's true state, forcing it to infer the user's beliefs, desires, and intentions (BDI) dynamically.

2. Latent Mental State Representation

MASP utilizes a BDI (Belief, Desire, Intention) based hybrid representation for the latent state $z_t$ :

Textual Components: Free-form natural language descriptions of Belief ( $B_t$ ), Desire ( $D_t$ ), and Intention ( $I_t$ ).
Scalar Components: Receptivity ( $\rho_t$ ), Commitment/Confidence ( $c_t$ ), and Valence ( $v_t$ ).
These components are encoded using a frozen text encoder and combined into a vector state. This representation allows for interpretable target state design while remaining general enough for diverse domains.

3. Teacher-Student Mentalization Architecture

A key design choice is the separation of mentalization into two roles to prevent policy exploitation:

Frozen Teacher Mentalizer ( $F_\omega$ ): Trained on offline "silver" annotations (LLM-generated BDI labels) and frozen during self-play. It provides the ground-truth latent state $z_t$ used for reward computation.
Student Mentalizer ( $M_\phi$ ): Conditions only on observable dialogue history to produce the estimate $\hat{z}_t$ used by the system policy.
This separation ensures the system learns to infer the user's state rather than directly exploiting the reward model's state estimator.

4. Adversarial Rewards and Objectives

The framework defines distinct reward functions for the system and the user to encourage robust behavior:

System Reward ( $r^S_t$ ):
$r^S_t = w_{prog}\Delta s_t - w_{inf}\|\hat{z}_t - z_t\|^2_A$
The system is rewarded for progress ( $\Delta s_t$ ) toward a task-specific target state relative to the user's initial state, and for accurate mentalization (minimizing the distance between its estimate and the teacher's state).
User Reward ( $r^U_t$ ):
$r^U_t = -w_{prog}\Delta s_t + w_{rat}q_t$
The user is trained to resist unjustified state changes (negative progress term) but is rewarded for rational behavior ( $q_t$ ). A fixed rationality judge determines if a user's resistance is coherent given their current mental state. This prevents the user from collapsing into unconditional rejection or unconditional agreement.

5. Training Pipeline

MASP follows a three-phase pipeline:

Phase 0: Train the Teacher and Student mentalizers using silver BDI annotations.
Phase 1: Warm-start the system and user policies using behavior cloning (supervised learning) on human dialogues.
Phase 2: Alternating GRPO (Group Relative Policy Optimization) self-play. The system and user policies are updated in turns while reward-side modules (Teacher, Judge) remain frozen. The student mentalizer is periodically refreshed using self-play prefixes.

Key Contributions

First BDI-Based Mentalized Training: To the authors' knowledge, MASP is the first multi-turn dialogue framework to explicitly model evolving BDI mental states as the basis for the training interaction, rather than just as an auxiliary feature.
Mentalized Self-Play: The paper introduces a self-play method that jointly trains the system and an adversarial user, specifically designed to avoid the "soft user" problem by making rational resistance a core training objective.
Rationality-Constrained Success (RCSR): The authors introduce a new evaluation metric, RCSR, which counts a dialogue as successful only if the task is completed and the user's behavior throughout the interaction remains rational. This filters out successes caused by simulator sycophancy.

Experimental Results

MASP was evaluated on four benchmarks: Persuasion for Good (P4G), ESConv (Emotional Support), EmpatheticDialogues, and CraigslistBargain (Negotiation).

Performance Gains: Across all datasets and backbone models (Qwen3-4B and Qwen3-0.6B), MASP substantially outperformed both Supervised Fine-Tuning (SFT) and "Planner-only" baselines (which use adaptive planning but no adversarial user training).
- Success Rate (SR): MASP achieved the highest SR in both soft and hard user conditions. For example, on P4G with Qwen3-4B, MASP achieved 0.859 SR (soft) and 0.788 SR (hard), compared to 0.616 and 0.576 for SFT.
- Efficiency: MASP reduced the Average Turns (AT) to success significantly.
- Rationality: MASP showed the highest RCSR, indicating its successes are grounded in rational user interactions rather than simulator compliance.
Robustness: In ablation studies and robustness tests against an ensemble of user simulators, MASP maintained high performance even in worst-case scenarios, whereas baselines dropped significantly.
Learning Dynamics: Analysis showed that MASP's gains emerge early in the dialogue (e.g., by turn 3-4) rather than being a result of late-turn recovery, suggesting the system learns to identify user states and make rational progress faster.

Significance and Claims

The paper claims that treating user resistance as a training objective rather than merely an evaluation condition is crucial for developing robust proactive dialogue agents. By modeling the dialogue as a partially observable game over a latent mental state, MASP forces the system to learn not just what to say, but who it is talking to and how that person is changing.

The authors emphasize that their approach addresses the brittleness of current policies trained against compliant simulators. The introduction of RCSR provides a more realistic measure of performance against difficult users. While the method relies on silver annotations and LLM-based judges (which introduces potential bias), the authors argue that the resulting policies are significantly more robust and strategically adaptive than those trained with standard planning or SFT approaches. The work suggests that future proactive dialogue systems must integrate opponent modeling and mentalization directly into the optimization loop to achieve genuine strategic capability.