Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue

1. Problem Statement

The paper addresses a critical gap in current Task-Oriented Dialogue (TOD) systems: the inability to effectively balance empathetic communication (user satisfaction) with budget-aware decision-making (operational costs) in real-world customer service scenarios.

Existing methods suffer from two primary limitations:

• Static Data Suboptimality: Most approaches rely on Supervised Fine-Tuning (SFT) using static dialogue corpora. This encourages models to imitate human behaviors, including cost-inefficient decisions (e.g., premature concessions or unnecessary compensation), without assessing long-term optimality or strategic trade-offs.
• Lack of Cost Modeling: Current frameworks typically optimize for a single metric like "success rate" or "user satisfaction." They fail to penalize "false successes"—resolutions achieved through excessive resource expenditure (e.g., unnecessary vouchers or protracted dialogues)—thereby ignoring the economic constraints of real-world deployment.

The core challenge is to train an agent that can resolve user issues, maintain service norms, and control operational costs simultaneously within a dynamic, non-cooperative environment.

2. Methodology: InteractCS-RL

The authors propose InteractCS-RL, a multi-granularity reinforcement learning framework designed to reframe TOD as a Constrained Markov Decision Process (CMDP). The framework consists of two main components:

A. User-Centric Interaction Framework

To overcome the limitations of static datasets, the authors construct a high-fidelity, dynamic training environment:

• Persona Profiles Bank: They extract features from real business logs to create a bi-level user profile:
  • Intrinsic Traits ($P_{int}$): Stable psychological dimensions including Communication Style, Information Disclosure, Problem-solving Style, and Personal Affective Style.
  • Extrinsic Demands ($D$): Goal-oriented patterns such as Rigid Pursuit, Preference-learning, Incentive-open, and Feedback-oriented.
• User Simulation: An LLM-based user simulator ($\pi_{user}$) is conditioned on these profiles and real-time business signals to generate diverse, realistic, and non-cooperative user interactions. This creates a closed-loop environment where the agent must adapt to varying user personalities and demands.

B. Cost-aware Multi-turn Policy Optimization (CMPO)

The core algorithmic contribution is a hybrid advantage estimation strategy that unifies three reward signals into a single learning objective:

1. Session-level Outcome Utility ($R_O$): A sparse reward based on the final user satisfaction score and task completion status.
2. Process Credit Assignment ($R_P$): To address the sparsity of terminal rewards, a Generative Reward Model (GenRM) evaluates every turn of the dialogue. It scores adherence to principles like empathy, logical consistency, and business compliance, providing dense, fine-grained feedback.
3. Cost-aware Lagrange Penalty: To handle global cost constraints (e.g., a maximum voucher issuance rate), the authors formulate the problem using a Lagrangian dual. Crucially, they introduce a PID (Proportional-Integral-Derivative) Controller to dynamically adjust the penalty multiplier ($\lambda$).
   • The PID controller stabilizes the training by correcting instantaneous violations (Proportional) and long-term bias (Integral), preventing the oscillation often seen in standard Lagrange multiplier methods.

The final objective function combines these elements to maximize utility while strictly adhering to cost constraints, effectively guiding the policy toward the Pareto boundary between service quality and economic efficiency.

3. Key Contributions

• Reframing TOD as Multi-Granularity RL: Moving beyond static trajectory imitation to a closed-loop interactive evolution cycle that balances turn-level quality, session-level outcomes, and global cost constraints.
• User-Centric Interaction Framework: A novel simulation environment driven by diverse persona profiles (intrinsic traits + extrinsic demands) that provides a realistic, high-fidelity training gym for agents.
• CMPO Algorithm: A hybrid optimization method integrating:
  • Generative Process Credits: Ensuring high-quality dialogue at every turn.
  • PID-Lagrangian Control: A robust mechanism for internalizing global budget constraints into the agent's decision-making process without sacrificing stability.
• Empirical Validation: Comprehensive experiments demonstrating that the method outperforms SFT, standard RL baselines (PPO, GRPO), and state-of-the-art closed-source models.

4. Experimental Results

The framework was evaluated on FoodDeliveryService (FDS), a realistic food delivery dispute scenario, and $\tau^2$-bench for cross-domain generalizability.

• Performance on FDS:

  • Task Effectiveness: The InteractCS-RL model (based on Qwen-2.5-14B) achieved a user satisfaction score of 3.05 in hard scenarios, surpassing top-tier closed-source models like GPT-4.1 (1.91) and LongCat-Flash (2.18) by nearly 40%. It also achieved a 100% dialogue finish rate.
  • Dialogue Quality: It scored highest in logical quality (11.34) and communication quality (27.43), showing a ~10% improvement over SFT baselines.
  • Cost Control: While other models (including SFT) exceeded the voucher rate threshold (often >40%), InteractCS-RL successfully maintained the voucher rate around the strict 30% threshold while maximizing satisfaction.

• Ablation Studies:

  • Removing the PID controller led to unstable cost control (oscillating voucher rates).
  • Removing Process Credits (GenRM) degraded logical consistency and communication quality.
  • Removing Cost Constraints resulted in high satisfaction but excessive budget overruns.

• Cross-Domain Generalizability:

  • On the $\tau^2$-bench (Retail, Telecom, Airline), InteractCS-RL consistently outperformed SFT and base models, improving Pass@1 rates by 5.6% on average for the 14B model. This indicates the framework learns transferable reasoning capabilities rather than just pattern matching.

5. Significance

This paper represents a significant step forward in deploying LLM-based agents for real-world business applications.

• Economic Viability: It demonstrates that it is possible to train agents that are not only helpful but also cost-conscious, preventing the "race to the bottom" where agents offer excessive compensation to satisfy users.
• Strategic Reasoning: By moving from static imitation to dynamic reinforcement learning with multi-granular rewards, the agents learn to navigate complex trade-offs between empathy, logic, and budget, mimicking the strategic decision-making of top human agents.
• Robustness: The integration of PID control for constraint handling offers a stable and scalable solution for safe RL in domains with strict operational boundaries, applicable beyond customer service to any domain requiring resource-constrained decision-making.