Technical Summary: Unlocking Proactivity in Task-Oriented Dialogue

1. Problem Definition

The paper addresses the challenge of Proactive Task-Oriented Dialogue (TOD), specifically in scenarios like outbound sales where an agent must initiate interactions, identify latent user concerns, and steer conversations toward a predefined goal (e.g., acceptance) within a bounded number of turns.

The authors identify two structural gaps in existing approaches:

1. Lack of Implicit Concern Modeling in Simulators: Current user simulators typically model only surface traits (tone, politeness), failing to capture the latent concerns that actually drive user decisions. This results in homogeneous trajectories and weakly grounded simulator decisions.
2. Inherent Reactivity of LLMs: Standard Large Language Models (LLMs) are aligned to be helpful and reactive, making them ill-suited for the initiative-taking required in proactive TOD. Post-training methods like Supervised Fine-Tuning (SFT) lack high-quality data, while Reinforcement Learning (RL) methods (e.g., GRPO) struggle to credit specific turns where persuasion occurs, often relying on sparse outcome rewards or generic quality scores rather than actual progress toward acceptance.

A pilot study demonstrates that while test-time scaling (Pass@16) and standard reward-shaping RL yield only marginal improvements, conditioning agents on users' latent concerns unlocks a significant jump in both proactivity and acceptance rates, moving agents from a "reactive plateau" to a high-performance regime.

2. Methodology

The proposed framework, SI-AVPO (Simulator-Induced Asymmetric-View Policy Optimization), consists of two core components designed to model latent concerns and convert them into training signals.

A. Cognitive User Simulator (CUS)

CUS is a stratified user simulator that models each user as a persona comprising three layers:

• Background ($P_{bg}$): Situational frames (e.g., demographics).
• External Persona ($P_{ext}$): Observable surface traits (tone, courtesy, communication style).
• Internal Persona ($P_{int}$): Hidden latent concerns across financial, psychological, and operational facets. Each concern includes a resistance point, unlock conditions, and anti-patterns.

CUS maintains a synchronous state dynamics system that tracks a scalar "willingness" state and the resolution status of each concern (Unresolved $\to$ Partially Addressed $\to$ Resolved). This produces:

• Privileged Signal: The internal persona $P_{int}$, invisible to the deployed agent.
• Simulation Feedback: Final binary decisions (Accept/Reject) and per-turn state transitions (willingness shifts $\Delta w$).

B. Simulator-Induced Asymmetric-View Policy Optimization

This module converts the signals from CUS into two complementary training objectives:

1. Asymmetric On-Policy Self-Distillation (AOPD):

   • Evaluates the same policy $\pi_\theta$ under two views: a privileged teacher view (conditioned on $P_{int}$) and a deployable student view (conditioned only on dialogue history).
   • Minimizes the KL divergence between the teacher's concern-aware distribution and the student's distribution.
   • Goal: Distill targeted, concern-aware strategies into the deployable policy without requiring a separate teacher model.

2. State-Transition Policy Refinement (STPR):

   • Uses the simulator's state dynamics to provide fine-grained credit assignment.
   • Trajectory-level Advantage: Derived from the final terminal reward ($r$).
   • Turn-level Credit: Adjusted by per-turn willingness shifts ($\Delta w$). If a turn's willingness shift aligns with the final outcome, the credit is reinforced; if contradictory, it is suppressed.
   • Goal: Guide the agent to balance initiative with context-aware responses, ensuring proactive moves are grounded in actual user receptiveness.

The total loss function is:
$$\mathcal{L}_{SI-AVPO} = \mathcal{L}_{AOPD}(P_{int}) + \lambda_{st} \mathcal{L}_{STPR}(d, \{\Delta w_k\})$$

3. Key Contributions

1. Cognitive User Simulator (CUS): A stratified persona framework calibrated on 100K+ real-world outbound call logs that explicitly models hidden internal concerns and synchronous state dynamics, producing faithful and behaviorally diverse interactions.
2. SI-AVPO Framework: A novel training paradigm that leverages information asymmetry (privileged $P_{int}$) and state transitions to create complementary objectives (AOPD and STPR) without external reward models, judges, or separate teacher models.
3. Empirical Validation: Demonstrated success on two real-world food-delivery benchmarks (Merchant Promotion and Courier Regional Bonus), showing robustness across different LLM-based user simulators.

4. Experimental Results

The method was evaluated on Qwen3-4B and Qwen3-8B backbones against:

• RL Baselines: GRPO, PPO, DAPO, and SEAD.
• Proprietary LLMs: GPT-4.1, DeepSeek-V3.2, GLM-5.1, Kimi-K2.5, and Claude-Sonnet-4.5.

Key Findings:

• Performance: SI-AVPO consistently outperforms all RL baselines. On the Merchant scene with Qwen3-8B, it improved Concern Solving Rate (CSR) by +12.2% and Acceptance Rate (Acc) by +12.2% over the best RL baseline (SEAD).
• Comparison with Flagships: SI-AVPO matches or surpasses leading proprietary models. With Qwen3-8B, it surpassed Claude-Sonnet-4.5 in CSR (52.6% vs. 51.1%) and Proactivity (3.18 vs. 3.09) on the merchant scene.
• Generalization: The method generalizes effectively across six different LLM-based user simulators (including strict ones like DeepSeek-V3.2), showing consistent improvements in CSR and Acc regardless of the simulator's inherent strictness.
• Ablation: Removing AOPD caused a sharp drop in Proactivity (-0.76), confirming its role in driving initiative. Removing STPR led to overly aggressive behavior, highlighting its role in grounding actions in context.

5. Significance and Claims

The paper claims that latent user concerns are the critical missing variable for proactive TOD. By explicitly modeling these concerns in a simulator and converting them into training signals via asymmetric views and state transitions, the authors demonstrate that:

• Proactive capability cannot be recovered merely through sampling or standard reward shaping; it requires a training-time signal grounded in the user's decision process.
• A deployable-size policy trained with SI-AVPO can match or exceed much larger, general-purpose proprietary models in proactive dialogue tasks.
• The framework provides a path to high-fidelity, cost-effective proactive agents without relying on external reward models or human judges.

The authors emphasize that their approach bridges the gap between reactive LLM alignment and the initiative-taking required for real-world outbound scenarios, offering a scalable solution for persuasive dialogue systems.