Corpus for Benchmarking Clinical Speech De-identification

To address the scarcity of publicly available resources for clinical speech de-identification, this paper introduces the SREDH-AICup corpus, a time-aligned dataset comprising 20 hours of English and Mandarin audio annotated with 7,830 sensitive health information entities across 38 categories to support automated privacy protection research.

The Gist

Imagine you have a giant, noisy library filled with millions of recorded doctor's visits. These recordings are goldmines for researchers trying to teach computers how to understand human speech. But there's a huge problem: these recordings are full of private secrets—patients' names, addresses, social security numbers, and specific dates of illness. If you just hand these recordings to a computer, it might accidentally learn those secrets and leak them later.

To fix this, we need a "privacy filter" that can listen to the audio, find the secrets in real-time, and silence them before anyone else hears them. But to build a good filter, you need a training manual. This paper is about creating that manual.

Here is the story of how the authors built the SREDH-AICup SHI speech corpus, explained simply:

1. The Missing Piece of the Puzzle

For years, researchers had two types of tools, but neither was perfect:

• Text Books: They had thousands of written medical notes where sensitive words were already crossed out and replaced with fake names (like "Patient A"). But these were just text, not audio.
• Audio Books: They had thousands of hours of people speaking, but these were mostly general conversations (like people ordering coffee or reading news) or medical talks that didn't have the "secret-crossing-out" labels attached to the specific moments in the audio.

The Gap: They needed a dataset that was both audio and had a precise map showing exactly when a secret word was spoken. It's like having a movie where the subtitles don't just say "Secret Name," but tell you exactly which second the actor said it, so you can mute just that split-second.

2. Building the "Privacy Gym"

The authors decided to build a "gym" where computers could practice their privacy skills. They combined three different ingredients to make a special recipe:

• Ingredient A (The Script): They took existing written medical notes (from a dataset called OpenDeID) where the secrets were already marked. They hired actors to read these scripts aloud naturally.
• Ingredient B (The Real Talk): They used recordings from a psychiatric dataset (DAMT) where doctors and patients were already talking.
• Ingredient C (The Drama): They grabbed clips from Taiwanese medical TV dramas. Why? Because real-life medical dramas often have doctors discussing patients, and they wanted to add some Mandarin Chinese to the mix to make it bilingual.

3. The "Millisecond" Hunt

Once they had the audio, they needed to label it. This wasn't a quick job.

• The Team: A group of trained annotators listened to the audio.
• The Task: They had to find 38 different types of secrets (like "Patient Name," "Hospital," "Phone Number," "Date of Birth").
• The Precision: They didn't just say, "The name was spoken." They had to mark the exact start and end time of the word down to the millisecond.
• The Analogy: Imagine a game of "Whac-A-Mole," but instead of hitting moles, they are hitting specific words in a conversation. If the mole (the secret word) pops up for 0.5 seconds, they have to hit it exactly within that 0.5 seconds. If they are off by even a tiny bit, the computer won't learn correctly.

They practiced this "hunting" game over and over (12 rounds!) until the team agreed on the timing 90% of the time. This ensured the map they created was incredibly accurate.

4. The Result: A 20-Hour Training Camp

The final product is a dataset of 20 hours of audio.

• The Breakdown: It's split into three piles: one for learning (training), one for checking progress (validation), and one for the final exam (testing).
• The Content: It contains about 7,830 "secrets" hidden inside the speech.
• The Languages: It's mostly English (about 19 hours), with a small but valuable slice of Mandarin Chinese (about 1 hour).
• The Reality Check: The data looks like real life. Some secrets (like "Date") appear constantly, while others (like "Passport Number") are rare. This "long tail" distribution is actually good because it mimics the messy reality of real hospitals, where some info is everywhere and some is rare.

5. Why This Matters

Think of this dataset as a flight simulator for privacy.
Before this, if you wanted to build a system that protects patient privacy in real-time (like a doctor speaking into a microphone while a computer transcribes it), you were flying blind. You didn't have a simulator to test your system.

Now, researchers can plug their new "privacy filters" into this dataset. They can see:

• Does the computer catch the name?
• Does it catch the phone number?
• Does it do it fast enough to stop the secret from leaking?

The Bottom Line

This paper is a gift to the scientific community. It provides the first high-quality, time-aligned "training ground" for teaching computers how to listen to medical conversations and instantly scrub out private information. It paves the way for a future where we can use AI to help doctors without ever worrying that the AI will accidentally reveal a patient's identity.

In short: They built a realistic, bilingual, high-precision "privacy obstacle course" so that future AI systems can learn to protect our most sensitive health secrets.

Technical Summary

1. Problem Statement

Despite significant advancements in Automatic Speech Recognition (ASR) and Named Entity Recognition (NER) for text, there is a critical scarcity of publicly available, time-aligned clinical speech datasets dedicated to de-identification.

• The Gap: Existing medical NLP datasets (e.g., i2b2, MIMIC-III, OpenDeID) are text-based. Conversely, large-scale speech corpora (e.g., LibriSpeech, Common Voice) are designed for general transcription and lack sensitive health information (SHI) annotations.
• The Challenge: Clinical speech de-identification requires not just transcribing speech but precisely locating and masking SHI entities (names, dates, IDs) within the audio stream in real-time. Current resources lack the necessary millisecond-level, time-aligned entity annotations required to train and evaluate models for this specific task, particularly in multilingual (English/Mandarin) clinical contexts.

2. Methodology

The authors constructed the SREDH-AICup SHI speech corpus by integrating and harmonizing three distinct data sources to create a unified, time-aligned dataset.

Data Sources & Integration

1. OpenDeID v2 (Text-to-Speech):
   • Source: 3,244 text-based Electronic Medical Records (EMRs) from the 2023 AICUP competition.
   • Process: 300 reports were randomly sampled, rephrased into natural clinical scripts by domain specialists (preserving original SHI structures), and recorded by 25 participants. This ensured privacy while creating speech data with ground-truth SHI labels.
2. DAMT (Direct Speech):
   • Source: High-quality psychiatric dialogue recordings from scripted clinical scenarios.
   • Process: Used directly as audio with existing transcriptions; split into training/validation sets.
3. PTS (Multilingual Expansion):
   • Source: Licensed medical drama segments from Taiwan Public Television Services.
   • Process: Medical dialogue segments were extracted, transcribed, and annotated to introduce Mandarin Chinese clinical speech, addressing the lack of non-English medical speech resources.

Annotation Schema & Process

• Schema: Based on the Health Science Alliance (HSA) guidelines, extended to include 38 subcategories across 8 primary SHI categories (e.g., NAME, PROFESSION, LOCATION, AGE, DATE, CONTACT, IDs, OTHER).
• Tools: Annotations were performed using Label Studio.
• Alignment:
  • Forced Alignment: Text and audio were synchronized using the Montreal Forced Aligner (MFA).
  • Segmentation: Voice Activity Detection (VAD) segmented recordings into clips under 30 seconds.
  • Temporal Boundaries: Annotators identified SHI spans with millisecond resolution start and end timestamps.
• Quality Control:
  • Five trained annotators participated.
  • Inter-Annotator Agreement (IAA): Evaluated using Fleiss' Kappa with a ±200ms temporal tolerance window (per NIST guidelines).
  • Calibration: After 12 iterative rounds on a subset of data, the Kappa score reached 0.907, exceeding the 0.8 threshold.

3. Key Contributions

• First Time-Aligned Clinical Speech Corpus: Introduces the first dataset specifically designed for speech-level de-identification with 38 fine-grained SHI categories.
• Multilingual Capability: Provides a bilingual dataset (English and Mandarin Chinese), addressing the severe shortage of Chinese medical speech resources.
• Standardized Benchmark: Establishes a reproducible benchmark with structured training (10h), validation (5h), and test (5h) splits, enabling fair comparison of de-identification models.
• Real-World Grounding: The dataset reflects the "long-tail" distribution of clinical documentation, including rare entities and structured identifiers, rather than artificially balanced data.

4. Results & Dataset Statistics

The final corpus comprises 20 hours of annotated audio with 7,830 SHI entities.

Metric	Value
Total Duration	20 hours
Total Files	3,024 (1,539 Train / 775 Val / 710 Test)
Language Distribution	19.36 hrs English (96.8%) / 0.89 hrs Mandarin (4.2%)
Source Contribution	59% DAMT, 36% OpenDeID v2 (re-recorded), 5% PTS
Audio Quality	Mean SNR > 28 dB across all subsets (Max 90.20 dB)
Entity Distribution	Long-tail distribution; most frequent: DATE (1,811), DOCTOR (1,365), PATIENT (828). Least frequent: URL (1), PHONE (2).

Note: The dataset exhibits a natural long-tail distribution where common entities (dates, names) dominate, while specific identifiers (phone numbers, URLs) are rare, mirroring real clinical documentation.

5. Significance and Impact

• Enabling Real-Time De-identification: The millisecond-level time alignment allows for the development of streaming de-identification systems that can mask sensitive data in real-time during live clinical conversations, moving beyond post-processing of transcripts.
• Privacy-Preserving AI: Facilitates the creation of robust privacy-preserving technologies for healthcare, ensuring compliance with regulations like HIPAA in voice-based AI applications.
• Multilingual Research: Highlights and partially addresses the gap in Chinese medical speech resources, encouraging further research into multilingual privacy protection.
• Benchmarking: Provides a standardized testbed for evaluating the performance of transformer-based ASR and NER models in complex, domain-specific clinical scenarios.

Conclusion: The SREDH-AICup corpus fills a vital void in medical AI research by providing a clinically grounded, time-aligned, and multilingual resource essential for advancing automated speech de-identification from theoretical text-based models to practical, real-world audio applications.