🤖 AI

A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks

This paper introduces MeDial-Speech, a novel 111+ hour speech dataset of robot-patient and doctor-patient medical dialogues covering four specific health conditions, along with a sentence selection benchmark that reveals state-of-the-art LLMs achieve moderate accuracy but exhibit significant overconfidence in their predictions.

Original authors: Heriberto Cuayahuitl, Grace Jang

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heriberto Cuayahuitl, Grace Jang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a kind, smart, and effective doctor. You can't just feed it a textbook; it needs to practice talking to people, hearing their worries, and responding with the right words. That is exactly what this paper is about: creating a massive "practice gym" for robots and AI doctors.

Here is the story of their new project, MeDial-Speech, broken down into simple parts:

1. The "Practice Gym" (The Dataset)

The researchers built a huge library of recorded conversations. Think of it as a gym for AI, but instead of lifting weights, the AI is lifting conversations.

The Size: They recorded over 111 hours of talking. That's like listening to a podcast non-stop for nearly five days straight.
The Players: They didn't just record real doctors and real sick people (which is hard to get permission for). Instead, they used actors.
- The "Patients": 325 volunteers (mostly students) played the role of people with specific health worries: memory loss (Lewy body dementia), heart trouble, shoulder pain, and chest pain (angina).
- The "Doctors": Medical students played the doctors.
- The "Robot": A robot named Pepper stood in the room. But here's the trick: the robot wasn't thinking on its own. A human "puppeteer" (a teleoperator) was in a different room, wearing headphones and watching a screen. When the robot spoke, it was actually the human doctor speaking through the robot's mouth. This is called a "Wizard of Oz" setup—like the Wizard in The Wizard of Oz pulling levers behind a curtain.

2. The "Magic Translator" (How it Worked)

The setup was a bit like a high-tech game of telephone:

The human doctor spoke into a microphone.
A computer instantly turned those words into text (Speech Recognition).
The computer added punctuation (like periods and commas) so it sounded natural.
The robot then spoke those words out loud to the "patient."
The patient answered, and the robot recorded everything.

This created a realistic loop where the "patient" thought they were talking to a robot, but the "robot" was actually a human doctor in disguise.

3. The "Exam" (Testing the AI)

The researchers didn't just collect the data; they put it to the test. They asked three super-smart AI brains (GPT-5 mini, DeepSeek-V3, and Claude Sonnet 4) to take a multiple-choice quiz.

The Game: The AI was given a snippet of a conversation and had to pick the best next sentence the doctor should say from a list of 20 options.
The Challenge: It was like a trivia game where 19 of the answers were random nonsense, and only 1 was the perfect medical response.
The Results:
- The Winner: Claude Sonnet 4 was the best student, getting about 71-75% of the answers right.
- The Problem: Even the winner (and the others) had a major personality flaw: Overconfidence.
- The Analogy: Imagine a student taking a test. If they get an answer right, they feel 100% sure. If they get it wrong, they still feel 100% sure. The AI models were like that. They didn't know when they were guessing. They were just as confident when they were wrong as when they were right.

4. Why This Matters (According to the Paper)

The paper claims this dataset is a free resource (for non-commercial use) to help researchers build better medical robots.

It helps train AI to understand how real conversations flow.
It helps test if robots can handle the "noise" of real life (like when a patient mumbles or speaks quietly).
It highlights that while AI is getting better at picking the right words, it still needs to learn humility—knowing when it doesn't know the answer.

Summary

Think of this paper as the release of a giant, free video game for AI developers. The game involves listening to hours of simulated doctor-patient chats. The researchers used it to show that while AI is getting good at playing the role of a doctor, it still acts like a confident fool—it thinks it's right even when it's wrong. The goal is to use this data to fix that confidence issue so that one day, robots can safely help humans with their health.

Technical Summary: MeDial-Speech Dataset and LLM Evaluation

1. Problem Statement

The application of Large Language Models (LLMs) to spoken medical consultations remains an open research challenge. While LLMs have advanced general-purpose AI, their deployment in interactive, spoken medical contexts is hindered by a scarcity of high-quality, publicly available datasets. Existing speech-based medical datasets are either closed-source, limited to human-human interactions, or rely heavily on synthetic data augmentation. Furthermore, current medical LLMs are primarily trained on text-based corpora, lacking specialization for the nuances of interactive, spoken dialogue involving both human-human and human-machine (robot-patient) interactions. There is a critical need for a free, accessible dataset that supports the development of conversational AIs capable of managing non-urgent medical conditions, thereby alleviating healthcare professional workloads.

2. Methodology

Data Collection System

The authors developed a "Wizard-of-Oz" (WOZ) system using a Pepper robot equipped with teleoperation capabilities.

Setup: A human doctor (teleoperator) was situated in a separate room from the robot and the patient. The teleoperator perceived the patient via audio/video streaming and conveyed responses through the robot.
Speech Processing: Instead of transmitting the teleoperator's raw audio, the system utilized continuous speech recognition (Vosk model) for real-time transcription, followed by automatic punctuation (FastPunct) and speech synthesis (Acapela). This ensured the patient heard synthesized robot speech rather than the human doctor's voice.
Non-Verbal Behaviors: The robot incorporated face tracking, head nodding, and randomized gestures (greeting, explaining, yes/no, please) to simulate natural interaction.
Recording: Audio was recorded at 16 kHz (16-bit, mono) for the robot/patient side and 44 kHz (stereo) for the TTS output. Video was captured at 10 fps (640x480).

Dataset Composition (MeDial-Speech)

Participants: 325 unpaid participants (mostly university students) acting as simulated patients.
Conditions: Dialogues covered four specific health conditions: Lewy body dementia, heart failure, shoulder pain, and angina.
Scale: The dataset comprises 111.4 hours of speech data, 581 dialogues, and approximately 264,451 words.
Annotations: Data includes manual transcriptions, speaker role labels (doctor, robot, patient, none), and segmentation files (Audacity). Approximately 70% of dialogues feature manual transcriptions for both doctor and patient utterances.
Format: Data is stored in separate folders per consultation, containing raw audio, text transcriptions, and segmentation metadata.

Benchmarking Protocol

To evaluate the dataset's utility for Spoken Language Processing (SLP), the authors proposed a sentence selection benchmark treating dialogue generation as a classification task.

Task: Given a dialogue context, an LLM must select the most appropriate next response from 20 options (one correct, 19 distractors randomly selected from unique doctor sentences).
Models Evaluated: GPT-5 mini, DeepSeek-V3, and Claude Sonnet 4.
Conditions:
- Without Noise: All utterances used manual transcriptions.
- With Noise: Patient utterances were replaced with Automatic Speech Recognition (ASR) transcriptions (generated via Faster-Whisper) to simulate real-world uncertainty, while doctor utterances remained manual.
Metrics: Balanced Accuracy, F1 Score, AUC, Brier Score, KL Divergence, Binary Cross Entropy, and Expected Calibration Error (ECE).

3. Key Results

LLM Performance

Best Performer: Claude Sonnet 4 outperformed the other models across most metrics.
- Without Noise: 71.19% Balanced Accuracy.
- With Noise: 74.73% Balanced Accuracy.
Noise Robustness: Claude Sonnet 4 and GPT-5 mini demonstrated robustness against ASR noise, whereas DeepSeek V3 showed performance degradation.
Statistical Significance: Signed tests on hard predictions indicated Claude Sonnet 4 was significantly better than the others ( $p < .05$ ). Wilcoxon Signed Rank Tests showed significant differences between Claude Sonnet 4 and DeepSeek V3 in soft predictions, though the difference between Claude and GPT-5 mini was not significant in noisy data.

Calibration and Overconfidence

A critical finding was the overconfidence of all evaluated LLMs.

Confidence Distribution: Models exhibited nearly equal confidence levels regardless of whether their sentence selection was correct or incorrect.
Overlap Coefficient (OVL): High overlap was observed between the Gaussian distributions of confidence scores for correct vs. incorrect predictions (OVL values ranging from 0.734 to 0.927).
Implication: The models fail to separate confidence levels between correct and incorrect predictions, indicating a need for improved model calibration to align predicted probabilities with reality.

4. Key Contributions

MeDial-Speech Dataset: The introduction of a novel, free-of-charge dataset containing 111+ hours of spoken medical dialogues from both robot-patient and doctor-patient interactions. It is unique in combining human-machine and human-human interactions with multiple annotation levels.
Realistic Data Collection: The use of a teleoperated robot system with continuous ASR and synthesis to create a realistic, interactive environment without relying on predefined GUI responses.
Dialogue Benchmark: A new evaluation framework using sentence selection (20-option multiple choice) to assess LLMs in medical dialogue contexts, specifically testing performance under noisy (ASR) conditions.
Empirical Analysis of Calibration: The study provides evidence that state-of-the-art medical LLMs are highly overconfident in their probabilistic predictions, highlighting a specific area for future model improvement.

5. Significance and Claims

The paper positions MeDial-Speech as a foundational resource for training and evaluating "Med-AIs" and automated clinicians capable of managing non-urgent conditions. By providing a dataset that bridges the gap between text-based LLM training and spoken, interactive medical reality, the authors aim to support:

Clinical Skills Training: Aiding medical students and trainees in communication and reasoning outside of clinical placements.
AI Development: Enabling the creation of conversational robots for healthcare delivery.
Research: Offering a benchmark for Spoken Language Processing tasks, including Voice Activity Detection (VAD), ASR, and Dialogue Generation.

The authors modestly conclude that while the dataset and benchmark reveal current limitations (specifically regarding model calibration and overconfidence), they provide a necessary step toward developing reliable, interactive medical AI systems. The dataset is released for non-commercial purposes to facilitate further research in this domain.