Technical Summary: DiPS – Dialogue Policy Selection for High-Stakes Persuasion Agents

Problem Definition

Large Language Models (LLMs) often struggle to execute effective persuasion in high-stakes scenarios, such as wildfire evacuations, where individual residents possess diverse personalities, concerns, and resource constraints. A "one-size-fits-all" approach frequently fails because residents vary drastically in their willingness to evacuate, their specific barriers (e.g., protecting property, caring for pets, logistical needs), and the strategies required to overcome them.

The core challenge is adaptive persuasion: an operator must dynamically tailor responses to a resident's evolving behavior under partial observability. The goal is to maximize the probability of a successful outcome (resident evacuation) by selecting the most appropriate high-level persuasion strategy at each turn of the dialogue, rather than relying on a fixed policy or generating responses without strategic planning.

Methodology: The DiPS Framework

The authors propose Dialogue Policy Selection (DiPS), a framework that treats persuasion strategy selection as an offline reinforcement learning (RL) problem. Instead of directly generating natural language responses, DiPS operates at a higher level, selecting a "persona" or policy from a set of candidate strategies to condition the response generation.

1. Problem Formulation

The interaction is modeled as a Partially Observable Markov Decision Process (POMDP):

• State ($s_t$): A latent state representing the resident's internal factors (beliefs, emotions, risk perception), approximated by sentence embeddings of recent resident utterances.
• Action ($a_t$): The selection of a discrete persuasion policy (persona) $z_t$ from a set $\Pi = \{\pi^{(1)}, \dots, \pi^{(K)}\}$. Each policy represents a distinct strategy (e.g., empathetic, directive, informational).
• Reward ($r_t$): A sparse reward signal where $r=1$ if the resident agrees to evacuate at the end of the dialogue, and $0$ otherwise.

2. Core Algorithm: Implicit Q-Learning (IQL)

To learn the optimal policy selection without requiring further environment interaction, DiPS employs Implicit Q-Learning (IQL).

• Offline Training: The system is trained on a static dataset of dialogues collected from human operators and simulated interactions.
• Value Function: A critic is trained to estimate the long-term value $Q(h_t, z)$ of selecting a specific persona $z$ given the dialogue history $h_t$.
• Mechanism: IQL avoids the distributional shift problem common in offline RL by using expectile regression to approximate the maximum over actions. It learns to assign higher values to personas that lead to successful outcomes in the training data while down-weighting those observed in failed trajectories.
• Policy Extraction: At inference, the agent selects the persona $z_t = \arg\max_z Q(h_t, z)$. This selected persona is then used to prompt an LLM to generate the actual operator response, often augmented with retrieved examples (RAG) specific to that persona.

3. Experimental Setup

The framework was evaluated in a wildfire evacuation simulation involving:

• Resident Profiles: Ten distinct personas (five from prior work, five new) with varying backstories and resistance levels (e.g., stubborn, cooperative, caregiver, logistical constraints).
• Baselines:
  • Zero-shot: LLM without prior data or strategy selection.
  • RAG (Global): LLM retrieving examples from a global pool of successful dialogues without policy selection.
  • Random Policy: Random selection of personas.
• Evaluation Environments:
  1. Simulation: LLM-based resident simulators.
  2. Human Subjects: Human participants role-playing as residents.
  3. Improved Simulation: Refined simulators with explicit barrier encoding to better mimic human resistance.

Key Results

1. Simulation vs. Human Discrepancy

In the initial experiments using standard LLM simulators, DiPS underperformed compared to Zero-shot and RAG baselines. However, when evaluated with human participants, DiPS achieved the highest overall success rate (79% vs. 74% for Zero-shot and 68% for RAG). This suggested that the initial simulation environment failed to capture the nuances of human resistance that DiPS is designed to address.

2. Improved Simulation Results

After refining the simulation environment to include explicit constraints (e.g., requiring concrete transport plans for specific characters), DiPS demonstrated superior performance:

• Success Rate: DiPS achieved 92% success, outperforming Zero-shot (88%) and RAG (78%).
• Efficiency: DiPS required fewer dialogue turns on average (11.2 turns) compared to baselines (11.8–11.9 turns), indicating more efficient persuasion.
• Generalization: DiPS maintained high performance on unseen resident profiles (96% on new residents in the improved setting), demonstrating its ability to generalize learned strategies to new behavioral profiles.

3. Ablation Studies

Removing components of DiPS consistently degraded performance:

• Replacing per-policy retrieval with global retrieval dropped success to 86%.
• Removing retrieval entirely dropped success to 80%.
• Using a mismatched policy (fixed strategy for all residents) resulted in a drastic drop to 51% success, highlighting the cost of non-adaptive strategies.

4. LLM-as-Judge Validity

The study utilized an LLM (GPT-4o-mini) as a judge to evaluate dialogue outcomes. While the LLM showed reasonable correlation with human annotators (72.2% agreement, $\kappa=0.42$), it was more conservative, often labeling conditional agreements as "maybe" rather than "yes." DiPS outcomes showed the highest agreement with human judgment (83.3%), suggesting its strategies produce clearer, more definitive outcomes.

Significance and Claims

The paper claims to present the first application of Implicit Q-Learning to high-stakes persuasive dialogue. Its primary contributions are:

1. Bridging Offline RL and Persuasion: It demonstrates that offline RL can effectively learn turn-level policy selection from heterogeneous dialogue data, moving beyond simple behavior cloning or static prompting.
2. Dynamic Adaptation: DiPS proves that dynamically switching between distinct persuasion personas based on conversational context is more effective than static strategies, particularly in high-stakes domains where resident resistance is nuanced.
3. Simulation Gap Identification: The work highlights a critical gap between LLM-based simulations and human behavior. It shows that standard simulators may mask the benefits of personalization, and that improved simulation design (explicitly encoding barriers) is necessary to accurately evaluate adaptive agents.
4. Practical Applicability: The framework offers a scalable method for deploying AI operators in disaster relief scenarios, where human operators are scarce and individualized communication is critical for saving lives.

The authors conclude that while LLMs have implicit persuasive abilities, explicit strategic planning via policy selection significantly enhances performance in complex, high-stakes interactions, provided the evaluation environment accurately reflects the constraints and resistance of real human behavior.