🤖 AI

CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants

This paper introduces CallBench, a comprehensive Chinese benchmark comprising 50,000 multi-turn dialogues across six scenarios to evaluate the challenging dual-goal coordination capabilities of phone call assistants, revealing that current methods struggle to effectively balance the device owner's explicit preset goals with the caller's implicit and dynamic objectives.

Original authors: Xuzhao Geng, Haozhao Wang, Xuelian Li, Zhenyu Yang, Haonan Lu, Rui Zhang, Ruixuan Li

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuzhao Geng, Haozhao Wang, Xuelian Li, Zhenyu Yang, Haonan Lu, Rui Zhang, Ruixuan Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the conductor of a very tricky orchestra. In the world of computer science, this is called building a "dialogue system"—a robot that can talk to humans. For a long time, scientists have taught these robots how to listen to a single person and help them finish a task, like booking a flight or ordering a pizza. It's like a robot playing a duet with one musician; they know the song, they know the goal, and they just need to hit the right notes. But what happens when the robot isn't just talking to one person? What if it's standing in the middle of a busy street, holding a phone for someone else, trying to listen to a stranger on the other end of the line while also remembering a secret list of instructions from the person who owns the phone? This is the messy, real-world challenge of a "proxy" setting. It's not just about finishing a task; it's about juggling two different people's needs at once without saying the wrong thing, revealing secrets, or making promises you can't keep.

This is exactly the problem a new study tackles. The researchers introduce a massive new testing ground called CALLBENCH, designed specifically to see how well AI assistants can handle these double-duty phone calls. They created 50,000 complete, multi-turn phone conversations in Chinese, covering six different scenarios like ordering food, catching a taxi, dealing with work calls, and even handling harassment. The goal was to see if current AI models could figure out when to stick to the phone owner's preset instructions (like "leave the package at the door") and when to pause those instructions because the person calling has a problem (like "the package is broken").

The study suggests that while current AI models are getting better at talking, they are still struggling with this specific "dual-goal" juggling act. The researchers found that existing methods often fail to balance the two goals, sometimes blindly pushing the owner's instructions even when the caller is in distress, or conversely, getting so distracted by the caller that they forget the owner's instructions entirely. They also discovered that safety is a huge hurdle; the AI often accidentally crosses the line into making unauthorized decisions or revealing private info.

To test this, the team didn't just look at whether the call ended successfully. They built a detailed scoring system that judges the AI on seven different levels: did it understand what was said? Did it remember the context? Did it guide the conversation at the right time? Was the response natural? Did it follow the owner's rules? Did it keep the conversation moving at a good pace? And most importantly, was it safe?

The results were a bit of a wake-up call. Even the smartest AI models tested, including some that are usually very good at planning and reasoning, scored surprisingly low on the overall test. The most common mistakes weren't just about sounding robotic; they were about bad timing and safety. For instance, the AI would often repeat the owner's preset message even when the caller had just reported an emergency, or it would try to end the call too quickly before all the details were sorted out. The study suggests that to build a truly reliable phone assistant, we need models that can make careful, turn-by-turn decisions about which goal to prioritize, all while staying strictly within the safety boundaries of a "proxy" who is just a messenger, not a decision-maker. It turns out that being a good phone assistant is less about having a big vocabulary and more about knowing exactly when to speak, when to listen, and when to stay silent.

Technical Summary: CALLBENCH

Problem Definition

The paper addresses a critical gap in target-oriented dialogue systems: the proxy setting inherent to phone call assistants. Unlike traditional systems that interact directly with a user to complete a single, explicit goal, a phone call assistant acts on behalf of a device owner. This creates a Dual-Goal Coordination problem where the system must simultaneously manage:

The Owner's Explicit Goal: Predefined instructions (presets) set by the device owner (e.g., "leave the package at the door").
The Caller's Implicit Goal: Dynamic, often unstated intentions that emerge during the conversation (e.g., reporting a spilled order or harassment).

These goals can be aligned, complementary, irrelevant, or conflicting. The core challenge is not merely completing a task but deciding when to advance, delay, or suspend the owner's preset based on the evolving caller situation. Furthermore, the system operates under strict safety boundaries: it must not disclose private information, fabricate facts, claim physical agency (e.g., "I can see the package"), or make unauthorized decisions (e.g., agreeing to a refund) on behalf of the owner.

Existing benchmarks (e.g., MultiWOZ, SGD) focus on single-user, single-goal interactions or general agent policy compliance, failing to capture the complexity of mediating between two independent parties under proxy constraints.

Methodology

Benchmark Construction (CALLBENCH)

The authors introduce CALLBENCH, a Chinese benchmark designed specifically for evaluating dual-goal coordination.

Scale and Scope: The dataset contains 50,000 complete multi-turn phone call dialogues across six scenarios: takeout, delivery, taxi, work, life, and harassment.
Preset Configuration: The benchmark models three distinct cases:
1. Regular Preset: High-frequency service calls (takeout, delivery, taxi) with predefined trigger conditions and owner responses.
2. Emergent Preset: User-defined conditional instructions that can trigger in any scenario based on specific caller utterances.
3. No-Preset: Cases where no owner-side goal is active.
Dual-Goal Relations: Dialogues are constructed to explicitly cover four relationship types between owner and caller goals: alignment, complementarity, irrelevance, and conflict.
Generation Process: A structured role-play generation pipeline using Doubao-Seed-1.6 creates the dialogues. One agent acts as the caller (driven by caller-side goals) and the other as the assistant (driven by owner presets and safety constraints). The process includes automatic quality checks and manual inspection (10% sampling) to ensure naturalness and safety compliance.

Evaluation Protocol

The paper proposes a preset-aware, turn-level evaluation protocol rather than relying solely on task success or surface-level fluency. An automated judge (Qwen3.7-Max) scores each assistant turn on a 0–1 scale across seven dimensions:

Semantic Understanding: Correctly interpreting the caller's current utterance.
Context Use: Leveraging dialogue history to avoid repetition or contradictions.
Active Guidance: Steering the conversation toward a state where the preset can be naturally conveyed without ignoring caller exceptions.
Response Quality: Naturalness, conciseness, and colloquial suitability for phone calls.
Preset Compliance: Conveying the owner's preset at the appropriate time (not too early, not too late, not during conflicts).
Dialogue Rhythm: Managing call progression and closing appropriately.
Safety: Strict adherence to privacy, decision-boundary, factuality, and non-embodiment constraints (scored as binary 0/1; any violation results in a 0).

Key Contributions

Task Formulation: The authors formalize the call assistant task as a turn-level decision process requiring dynamic coordination of explicit owner targets and implicit caller intentions under strict safety constraints.
Dataset Construction: The creation of CALLBENCH, the first large-scale (50k dialogues), multi-scenario benchmark for dual-goal coordination in phone call assistants, covering diverse preset cases and goal relations.
Evaluation Framework: The design of a fine-grained, multi-dimensional evaluation protocol that assesses not just what the assistant says, but when and why it says it, specifically regarding preset timing and safety.
Empirical Analysis: A comprehensive evaluation of representative task- and target-oriented dialogue frameworks (e.g., ReAct, SimpleTOD, AutoTOD) to quantify bottlenecks in dual-goal trade-offs.

Experimental Results

The authors evaluated eight representative methods (including Direct Prompting, ReAct, SimpleTOD, DivTOD, AutoTOD, ProCoT, EnPL, and ChatSOP) using Doubao-Seed-1.6 as the backbone. Key findings include:

Performance Gap: Existing methods do not transfer directly to the phone call assistant setting. While some methods perform well on specific dimensions (e.g., semantic understanding), they struggle to balance goal coordination, dialogue rhythm, and safety simultaneously.
Top Performer: ReAct achieved the strongest average performance across both LLM-judge and human evaluation settings, though absolute scores remained far from saturated.
Error Analysis: The most frequent errors were safety issues (10.63% average rate), followed by repetitive replies, improper closing, and preset non-compliance (over-following or repeating presets). This indicates that current models fail primarily due to poor turn-level decision-making under dual-goal conditions rather than surface-level fluency issues.
Correlation: There is a strong positive correlation (Kendall's $\tau_b = 0.764$ ) between automatic LLM-judge scores and human ratings, validating the evaluation protocol.

Significance and Claims

The paper claims that CALLBENCH reveals critical weaknesses in current dialogue systems that are overlooked by traditional benchmarks. The significance lies in demonstrating that effective phone call assistants require reliable turn-level decision-making to navigate the tension between two independent goals under strict proxy constraints.

The authors modestly state that their work highlights the need for new approaches capable of:

Dynamically coordinating explicit and implicit goals.
Adhering to strict safety boundaries (privacy, non-embodiment, no unauthorized decisions).
Managing dialogue rhythm and preset timing in noisy, evolving contexts.

The paper concludes that CALLBENCH serves as a necessary tool to guide future research toward building reliable, safe, and context-aware phone call assistants for real-world deployment.