When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue

Technical Summary: When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue

Problem Statement

Task-oriented dialogue (TOD) systems, which assist users in booking hotels, finding restaurants, and arranging transport, typically rely on a pipeline where Large Language Models (LLMs) extract intent, query a backend database, and generate responses grounded in the results. This pipeline is fragile at the database boundary. In real-world deployments, backend failures—specifically empty results (no matching records), wrong-domain returns (data from an incorrect domain), and API errors (timeouts or server failures)—often cause LLM agents to produce fluent but unsafe responses. Instead of acknowledging the failure, agents frequently hallucinate, inventing venues, confirmations, or booking details not supported by the database. Prior work has largely focused on general agent tool failures or user-behavior stressors, lacking a controlled evaluation of runtime database execution faults in TOD systems that isolates prompting-only recovery strategies without retraining.

Methodology

The authors propose a controlled fault-injection framework to evaluate recovery strategies across two structurally different datasets: MultiWOZ 2.2 (5 domains) and SGD (20 domains).

• Fault Injection: For each test dialogue, the system extracts the first user turn and synthetically injects one of four database conditions:

  1. Clean: Valid result (sanity baseline).
  2. Empty Result: Zero matches (correct action: acknowledge failure, offer constraint relaxation).
  3. Wrong Domain: Results from an incorrect domain (correct action: detect mismatch, confirm with user).
  4. API Error: Timeout or 503 error (correct action: apologize, ask to retry).
     Note: Using the first turn isolates immediate recovery behavior, avoiding confounds from multi-turn state tracking.

• Recovery Strategies: The study compares three prompting strategies across six open-weight model families (DeepSeek-R1, Gemma-2, Llama-3, Mistral, Phi-3, Qwen-2.5) using a fixed decoding temperature (0.2):

  1. Naive: Raw database response passed with no special instruction.
  2. Inform: Explicit instruction not to hallucinate and to acknowledge failures honestly.
  3. Guided-Retry: A structured decision procedure keyed on the database status field, providing explicit per-failure instructions and a strict prohibition on fabricating venue names, booking references, or confirmation details.

• Metrics:

  • Hallucination Rate (HR): Fraction of non-clean turns where the agent fabricates a result absent from the DB response.
  • Appropriate Action Rate (AAR): Fraction matching ground-truth correct actions per failure type.
  • Commitment Safety Rate (CSR): Fraction of non-clean turns free of explicit false commitments (e.g., invented booking refs). This metric was validated by human annotators (Fleiss' $\kappa = 0.7672$).
  • User Friction Score (UFS): A composite score penalizing hallucinations and silent failures.

Key Results

The study finds that while structured prompting significantly improves robustness, it does not eliminate hallucination.

• Overall Performance: The Guided-Retry strategy consistently outperforms both Naive and Inform baselines across all six models.

  • On MultiWOZ, Guided-Retry reduces the average Hallucination Rate (HR) from 30.5% (Naive) to 15.3% (a 50% reduction) and improves the Appropriate Action Rate (AAR) to 83.0%.
  • On SGD, HR is reduced from 20.9% to 12.2% (a 42% reduction), with AAR reaching 83.9%.
  • These improvements are statistically significant ($p < 0.001$) across all models.

• Model Variance:

  • DeepSeek-R1 achieved the lowest HR under Guided-Retry (5.8% on MultiWOZ).
  • Phi-3 was a notable exception; it performed worse under Guided-Retry than under the Inform strategy (HR 35.2% vs. 31.0%), suggesting it does not reliably follow structured recovery instructions.

• Failure Type Analysis:

  • Wrong-domain retrieval proved to be the most difficult failure case. Even with Guided-Retry, residual hallucination remained high (20.8% on MultiWOZ, 17.0% on SGD), indicating models often adopt vocabulary from the incorrect domain rather than detecting the mismatch.
  • API errors showed the clearest benefit from structured prompting, with HR reductions of roughly 25 percentage points on both datasets.

• Residual Hallucination: Despite improvements, a substantial residual hallucination rate of 6–37% remains across models and datasets, indicating that prompt-level recovery alone is insufficient for fully robust deployment.

Significance and Contributions

The paper makes four primary contributions:

1. Controlled Framework: A fault-injection framework for MultiWOZ and SGD that isolates runtime database execution faults from user-behavior stressors.
2. Prompting Evaluation: An evaluation demonstrating that a single structured system-prompt addition can reduce hallucination by 42–50% without retraining, additional inference calls, or significant engineering overhead.
3. Metrics: A set of automatic and human-validated metrics for failure recovery, including the Commitment Safety Rate (CSR).
4. Empirical Findings: The finding that while structured prompting reduces hallucination, it leaves a non-trivial residual rate (6–37%), with wrong-domain failures being the most persistent challenge.

The authors conclude that while Guided-Retry yields the safest behavior overall and is a practical, low-cost intervention for improving TOD robustness, it is not a complete solution. The persistence of residual hallucinations suggests that prompt-level recovery must be complemented by other measures for robust real-world deployment.