INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection

1. Problem Statement

Administrative tasks in U.S. healthcare, specifically insurance benefit verification, generate approximately $1 trillion in annual costs and involve over 500 million manual calls annually. While Large Language Models (LLMs) and voice agents are increasingly deployed to automate these workflows, current evaluation benchmarks fail to address the specific needs of regulatory auditing.

Existing NLP benchmarks focus on generic metrics like relevance, helpfulness, or turn-level intent recognition. They do not test whether a conversation contains ordered, phase-aligned evidence required for compliance (e.g., verifying patient identity before discussing coverage). Auditors require deterministic proof that specific checklist items occurred in a specific sequence with explicit "ask/answer" roles. Current models often fail to localize these events precisely or adhere to strict workflow gating rules (e.g., if a patient record is not found, downstream drug checks must be skipped).

2. Methodology

A. The INSURE-Dial Dataset

The authors introduce INSURE-Dial, the first public benchmark designed for phase-aware compliance evaluation in insurance benefit verification.

• Composition:
  • 50 Real Calls: De-identified, AI-initiated calls with live insurance representatives (mean 71 turns/call). These were manually annotated by human experts.
  • 1,000 Synthetic Calls: Generated using the Dataframer platform to expand coverage of edge cases (e.g., inactive plans, single vs. two-drug workflows) while maintaining the same structural constraints as real calls.
• Annotation Schema:
  • Calls are structured into a phase-aware JSON schema covering specific audit checkpoints: IVR, Greeting, Patient Identification (PID), Coverage Status Verification (CSV), Drug Loop (Formulary, Restrictions, Copay for up to two drugs), and Agent Interaction (CRN).
  • Gating Logic: The schema enforces deterministic workflow rules (e.g., if record_found=false, then CSV and Drug phases are marked "NA").
  • Compliance Labels: Each phase is labeled for:
    • Information Compliance (IC): Was the required information explicitly provided?
    • Procedural Compliance (PC): Was the information requested before being provided (Ask-then-Answer pattern)?
  • Span Segmentation: Every phase is annotated with an inclusive turn span $[i, j]$.

B. Evaluation Tasks

The benchmark defines two distinct tasks to isolate segmentation errors from reasoning errors:

1. Task 1: Phase Boundary Detection: The model must identify the start and end turns (spans) for each phase in the transcript.
   • Metrics: Exact Match (EM), Turn-level F1, Sum of Absolute Differences (SAD), and Call-level EM (all phases correct).
2. Task 2: Compliance Verification: Given fixed (ground truth) spans, the model must determine if IC and PC conditions are met based on explicit evidence.
   • Metrics: Phase-level accuracy, Macro-F1, and Call-level Hit (fraction of applicable phases correct).

C. Synthetic Data Generation Pipeline

To overcome the scarcity of regulated data, the authors used a Dataframer pipeline:

1. Analysis: An LLM infers constraints and variation axes from a small seed set of real calls.
2. Generation & Evaluation: A generator creates candidate calls, and a critic/evaluator (LLM) + deterministic checker validates them against the schema and gating rules.
3. Validation: Only samples passing a deterministic checker (verifying spans, roles, and gating logic) are accepted.

3. Key Contributions

1. First Public Benchmark for Audit-Grade Voice Agents: Unlike general dialogue datasets, INSURE-Dial focuses on ordered, evidence-based compliance with explicit span boundaries.
2. Novel Evaluation Framework: It decouples segmentation (finding where an event happened) from compliance reasoning (deciding if it happened correctly), allowing researchers to pinpoint specific failure modes.
3. Hybrid Corpus: A combination of 50 high-quality human-annotated real calls and 1,000 structurally validated synthetic calls, enabling robust stress-testing of models.
4. Open Release: The dataset, annotation schema, prompts, and scoring code are publicly available.

4. Results

The authors evaluated various LLMs (including GPT-4o-mini, GPT-4.1, and Gemini 2.5 Flash) on both tasks.

• Task 1 (Boundary Detection) - The Bottleneck:
  • Models achieve moderate phase-level accuracy (e.g., Gemini 2.5 Flash: ~72% EM on real calls).
  • Call-level Exact Match is near-zero (max 4.0% on real calls).
  • Reasoning: Because a call-level success requires all phases (8–10 per call) to be perfectly segmented, a single boundary error (even a 1-turn drift) causes the entire call to fail. This highlights that segmentation is the primary bottleneck for audit-grade extraction.
• Task 2 (Compliance Verification) - Strong Reasoning:
  • When provided with correct spans, models perform exceptionally well.
  • Top models achieve >93% phase-level accuracy and 88–96% call-level accuracy for IC/PC decisions.
  • This indicates that current LLMs can reliably apply compliance logic if the evidence is correctly localized.
• Real vs. Synthetic:
  • Synthetic data preserves model rankings and error patterns observed in real data but shows higher segmentation scores due to the absence of real-world noise (e.g., hold times, repetitive IVR prompts).
  • Synthetic data is effective for stress-testing edge cases (e.g., inactive plans) that are rare in the small real subset.

5. Significance and Implications

• Audit-Grade Reliability: The paper demonstrates that while LLMs are fluent in conversation, they currently lack the precision required for regulatory auditing. A "successful" call in terms of dialogue flow does not equate to a "compliant" call in terms of audit evidence.
• Workflow Gating is Critical: The study emphasizes that compliance is not just about content but about order and preconditions. Models must understand that if a patient record isn't found, they must not attempt to verify coverage.
• Future Directions: The authors suggest that future systems need better active learning for boundary correction and tighter integration of deterministic rules with neural models to bridge the gap between conversational fluency and audit-grade reliability.
• Ethical & Privacy: The dataset adheres strictly to HIPAA Safe Harbor de-identification, releasing only text transcripts (no audio) to ensure privacy while enabling research into regulated voice automation.

In conclusion, INSURE-Dial provides a critical tool for the healthcare AI community, shifting the focus from generic dialogue quality to verifiable, phase-structured compliance, and revealing that span boundary detection remains the most significant hurdle for deploying autonomous agents in regulated environments.