🤖 machine learning

Dual-Agent Co-Training for Health Coaching via Implicit Adversarial Preference Optimization

This paper proposes a dual-agent co-training framework that interactively optimizes both an AI health coach and a client simulator through implicit adversarial preference optimization, effectively overcoming the limitations of one-sided training to improve the quality and scalability of motivational interviewing-based health coaching.

Original authors: Da Long, Lingyi Fu, Diya Michelle Rao, Jasmine Ruales Carrera, Yang Bai, Shandian Zhe

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Da Long, Lingyi Fu, Diya Michelle Rao, Jasmine Ruales Carrera, Yang Bai, Shandian Zhe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Finding a Coach is Hard

Imagine you want to change a habit, like eating better or exercising more. The best way to do this is often with a "motivational interviewing" coach. These aren't just people who yell "Go!" at you; they are skilled listeners who help you find your own reasons to change.

However, real human coaches are expensive and hard to find. We need an AI coach that is cheap and available 24/7. But here's the catch: most AI coaches today are trained like a student practicing with a fixed textbook. The AI learns to talk to a "client" that never changes its mind or gets upset. It's like a tennis player practicing against a wall that always hits the ball back the same way. They get good at hitting that specific ball, but they crumble when a real, unpredictable human shows up.

The Solution: The "Sparring Partner" Method

The authors of this paper created a new training system called DACT (Dual-Agent Co-Training). Instead of training the AI coach against a static wall, they trained two AIs at the same time:

The Coach AI: The one we want to become a great health coach.
The Client AI: A simulator that plays the role of the patient.

Think of this like a martial arts dojo.

The Coach is the student trying to learn perfect technique.
The Client is the sparring partner.

In traditional training, the sparring partner stands still. In this new method, the sparring partner is learning too. Every time the Coach gets better at handling a specific type of resistance, the Client AI learns to throw a harder punch or a trickier move that the Coach hasn't mastered yet.

How They Train: The "Tree of Choices"

To teach these AIs, they don't just have a simple conversation. They grow a Decision Tree.

Imagine the Coach says something. Instead of just one reply, the system generates three different possible replies from the Coach. Then, for each of those, the Client generates three different possible reactions. This creates a branching tree of possibilities.

At the end of every branch, a Super-Referee (a powerful AI judge) looks at the conversation and scores the Coach on three specific skills:

Cultivating Change Talk: Did the Coach help the client find their own motivation?
Softening Sustain Talk: Did the Coach handle the client's resistance or excuses without arguing?
Empathy: Did the Coach truly listen and understand the client's feelings?

The Secret Sauce: "Pareto" and "Adversarial" Training

This is where the magic happens. The paper uses two clever tricks to make the training intense and effective.

1. The "No-Compromise" Rule (Pareto Dominance)
Usually, an AI might get really good at one thing (like being nice) but bad at another (like asking the right questions). The authors say: "No."
They only let the Coach learn from examples where it was better at everything at once. If the Coach's new reply is better at empathy and better at asking questions and better at handling resistance, it wins. If it's better at one thing but worse at another, it doesn't count. This forces the Coach to become a well-rounded expert, not a one-trick pony.

2. The "Reverse Psychology" Client
This is the "Adversarial" part. The Client AI is trained with reverse goals.

The Coach wants to get a high score.
The Client wants to make the Coach get a low score.

If the Client says something that confuses the Coach or makes the Coach argue (which lowers the Coach's score), the Client gets a "reward" for doing so.

The Result: The Client learns to be the ultimate difficult patient. It learns to be emotional, resistant, or ambivalent in ways that specifically expose the Coach's weaknesses.
The Benefit: As the Coach gets better at handling these difficult situations, the Client automatically shifts to finding new weaknesses to exploit. It's a self-driving curriculum that gets harder exactly as fast as the Coach can handle it.

The Results: A Coach That Can Handle Anything

The authors tested their new Coach against:

Standard AI Coaches: Trained on fixed data.
Super-Prompted AI: A powerful AI given a very long, detailed instruction manual on how to be a coach.
The "Tough" Client: A specific client simulator that was trained to be the hardest opponent possible.

The Findings:

The DACT Coach significantly outperformed everyone else.
It was much better at handling the "Tough Client" without breaking character or giving bad advice.
Most importantly, it made far fewer clinical mistakes (like arguing with the client or giving unsolicited advice) compared to the other methods.

The Bottom Line

The paper claims that by training the Coach and the Client together, where the Client constantly tries to "break" the Coach, the Coach learns to handle the messy, difficult, and emotional reality of real human conversations much better than if it were trained alone. It turns the training process into a dynamic game of "chess" where both players get smarter, resulting in a Coach that is ready for the real world.

Technical Summary: Dual-Agent Co-Training for Health Coaching via Implicit Adversarial Preference Optimization

Problem Statement

Motivational interviewing (MI) is a proven approach for improving mental health and promoting behavior change, yet its scalability is limited by the scarcity of trained human coaches and high service costs. While AI assistants exist, existing training methods for health coaching typically optimize only one side of the interaction: either training a dialogue agent against a fixed client environment or training a client simulator against a fixed assistant. This asymmetric setup restricts the exploration of the interaction space and fails to efficiently develop the complex, context-sensitive capabilities required for high-quality coaching. Furthermore, success in MI depends on the quality of behavior throughout the entire session, not just the final outcome, requiring optimization across multiple dimensions (e.g., cultivating change talk, softening sustain talk, empathy) rather than a single scalar reward.

Methodology: DACT

The authors propose DACT (Dual-Agent Co-Training), a framework that jointly trains a health coach agent and a client simulator through interactive dialogue. The core innovation is an implicit adversarial training dynamic where the client learns to generate increasingly challenging interactions, forcing the coach to improve.

1. Preference Dialogue Tree Construction

Instead of relying on expensive expert demonstrations, DACT constructs preference trees through a rollout procedure:

Initialization: A dialogue tree is initialized with a random client persona.
Branching: At selected dialogue states, the coach samples multiple candidate utterances. For each coach utterance, the client independently samples multiple responses.
Expansion: This branching-and-rollout process continues recursively to a predefined depth, creating a balanced tree where sibling nodes (utterances at the same depth) are compared under equal rollout budgets.

2. Multi-Dimensional LLM Judge and Q-Value Estimation

To evaluate coaching quality without human experts, the authors employ a multi-dimensional LLM judge (based on the MITI 4.2.1 coding manual) that scores coach utterances on three dimensions:

Cultivating Change Talk (CCT)
Softening Sustain Talk (SST)
Empathy

The judge operates in three stages: (1) identifying the client's state (e.g., engaged, ambivalent, emotional), (2) labeling the function of each sentence in the coach's response (e.g., open question, reflection, unsolicited advice), and (3) aggregating reasoning-based scores.

Q-Value Propagation: Immediate turn-level scores are propagated upward through the tree using a discounted recursive backup to estimate the long-term value ( $Q$ ) of a coach utterance, accounting for downstream effects on the dialogue trajectory.

3. Implicit Adversarial Preference Training

The framework uses Direct Preference Optimization (DPO) for both agents, but with opposing objectives derived from the same Q-values:

Coach Optimization: The coach is trained on Pareto-dominant preference pairs. A coach response $A$ is preferred over $B$ if $A$ 's Q-vector weakly dominates $B$ 's on all dimensions and strictly dominates on at least one. This ensures the coach improves across all dimensions simultaneously without trade-offs.
Client Optimization: The client is trained adversarially. Its preference signal is the negation of the coach's Q-values ( $S(u) = -Q(u)$ ). The client is trained to prefer utterances that result in lower downstream coaching quality (i.e., smaller Q-values for the coach). This induces the client to learn "harder" MI behaviors (e.g., strong resistance, complex ambivalence, emotional distress) that challenge the current coach's weaknesses.

4. Stochastic-Game Interpretation

The authors demonstrate that this co-training process admits a natural stochastic-game interpretation. The coach and client can be viewed as two players in a game with opposing latent utilities: the coach seeks to maximize downstream multi-dimensional coaching quality, while the client seeks to minimize it. The alternating DPO updates correspond to KL-regularized policy-improvement steps for these adversarially coupled players.

Key Contributions

Dual-Agent Preference Tree Construction: An efficient, low-cost method for collecting preference pairs during training using tree rollouts and a multi-dimensional LLM judge, avoiding reliance on expert demonstrations.
Multi-Dimensional LLM Judge: A clinically grounded evaluation system that decomposes judgment into state identification, functional labeling, and reasoning-based aggregation to estimate Q-values across CCT, SST, and Empathy.
Implicit Adversarial Preference Training: A novel training dynamic where the client is trained to reverse the coach's preferences, creating a self-paced adversarial curriculum that exposes the coach to increasingly difficult conversational situations without manual difficulty scheduling.
Stochastic-Game Formalization: A theoretical interpretation of the co-training dynamics as a two-player stochastic game with opposing utilities.

Experimental Results

The authors evaluated DACT against three baselines: a Supervised Fine-Tuning (SFT) baseline, a GPT-5.5 baseline with standard prompting (GPT-COACH), and a GPT-5.5 baseline with the scoring rubric embedded in the prompt (GPT-RUBRIC).

Performance: DACT achieved the highest overall coaching quality (mean3 score) across four client conditions, including an in-distribution "stress test" client (R8-CLIENT) and three out-of-distribution simulators (Emotional, Resistant, Ambivalent). On the challenging R8-CLIENT, DACT scored 4.25, significantly outperforming the GPT-RUBRIC baseline (3.33).
Anti-Pattern Suppression: DACT reduced the rate of MI anti-patterns (clinically undesirable behaviors) by 20–40 times compared to the next-best method. For instance, on the R8-CLIENT, DACT had an anti-pattern rate of 0.41%, whereas GPT-RUBRIC had 16.15%.
Co-Evolution Dynamics: Ablation studies showed that co-evolution is critical. When the client was frozen (Client-Frozen), the coach's performance plateaued early. Only with the adaptive client did the coach continue to improve, particularly in CCT and SST dimensions.
Human Alignment: A blind ranking study with a trained MI practitioner showed strong agreement (77.5%) between the LLM judge's preferences and human expert rankings, validating the LLM judge as a proxy for expert evaluation.

Significance and Claims

The paper claims that DACT effectively addresses the limitations of one-sided training by creating a self-improving loop where the coach and client co-evolve. The significance lies in:

Scalability: Providing a method to train high-quality AI health coaches without the prohibitive cost of human expert demonstrations.
Robustness: The co-training process produces coaches that generalize well to difficult, out-of-distribution client behaviors and suppress clinically harmful anti-patterns.
Theoretical Insight: The framework bridges preference optimization and game theory, showing that implicit adversarial dynamics can be formalized as a stochastic game.

The authors remain modest, noting that results are based on a single base model and that the magnitude of improvement on in-distribution stress tests may vary under full expert MITI evaluation. They identify replication on additional base models and full expert evaluation as necessary next steps.