Technical Summary: DialogPII

Problem Statement

Conversational data collected in domains such as healthcare, social sciences, and customer support is a critical resource for research and automated analysis. However, the responsible sharing of this data is hindered by the presence of personally identifiable information (PII). While existing research on text anonymization has primarily focused on formal medical documents (e.g., clinical notes), conversational language presents unique challenges. Spoken interactions involve fragmented sentences, informal language, interruptions, speaker changes, and spontaneous disclosures that make direct transfer from document-oriented de-identification frameworks difficult.

Furthermore, there is a significant scarcity of multilingual resources for conversational de-identification. Existing benchmarks are largely monolingual, English-centric, or focused on general web text rather than sensitive dialog scenarios. Real-world conversational datasets are difficult to share due to ethical and legal constraints regarding privacy. Consequently, there is a need for a high-quality, multilingual, synthetic dataset that covers diverse interaction scenarios and includes both written and speech-derived transcripts to support the development and evaluation of automatic de-identification systems.

Methodology

The authors present DialogPII, a dataset created through a five-step pipeline designed to generate synthetic, multilingual, and speech-aligned resources.

1. Annotation Schema Definition: The authors developed a dialog-oriented schema combining concepts from Personal Health Identifier (PHI) frameworks and the Text Anonymization Benchmark (TAB). This schema covers 19 entity types, including persons, organizations, locations, identification codes, professions, social relationships, products, temporal information, and quantities. The schema was iteratively refined to capture information that could contribute to re-identification in conversational settings.
2. Synthetic Dialog Generation: Dialogs were generated using the Gemini large language model (2.5 Pro) across eight interaction scenarios: emergency calls, medical anamnesis interviews, therapy sessions, insurance communication, customer support, clinical interviews regarding an AI-supported dashboard, police reports, and group therapy discussions. To ensure diversity, prompts specified speaker personas, geographic contexts (localized to specific cities in the UK, Germany, etc.), and interaction goals.
3. Curation and Annotation: Generated dialogs were manually curated to improve plausibility and remove repetitive artifacts. Sensitive information was annotated by two independent annotators using the INCEpTION platform. Disagreements were resolved by a third annotator, resulting in gold-standard annotations with a Krippendorff's alpha of 0.87.
4. Translation and Localization: The curated English dialogs were translated into ten additional languages (Arabic, Finnish, French, German, Hindi, Italian, Polish, Portuguese, Spanish, and Turkish) using the Gemini model. Crucially, these translations were localized to country- and city-specific contexts (e.g., adapting London settings to Berlin for German) to ensure cultural and linguistic realism. Annotations were projected onto the translated texts and manually corrected.
5. Verbalization and Transcription: All written dialogs were converted to synthetic speech using Google Cloud Text-to-Speech (TTS) APIs. These audio files were then transcribed using a locally deployed Whisper (large v3) model with Pyannote speaker diarization. This process introduced realistic ASR artifacts (transcription errors, punctuation inconsistencies) and speaker segmentation challenges. The original annotations were automatically projected onto these transcripts and manually curated to align with the speech-derived text.

Key Contributions

• Multilingual Dataset: DialogPII provides aligned written and speech-derived transcripts in 11 languages, covering 19 entity types and 8 distinct interaction scenarios.
• Speech-ASR Pipeline: Unlike previous works focusing solely on text, this dataset includes speech-derived transcripts generated via TTS and ASR, enabling research on de-identification in automatic speech recognition pipelines.
• Scenario Diversity: The dataset covers a broad spectrum of privacy-sensitive contexts, from emergency calls and police reports to therapy sessions and insurance communications, including extended multi-speaker group therapy conversations.
• Baseline Models: The authors release baseline multilingual Named Entity Recognition (NER) models trained on the dataset, utilizing a ModernBERT architecture with a CRF decoding layer and document-level context enrichment.
• Technical Validation: The paper provides comprehensive validation, including inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and benchmark experiments.

Results

• Annotation Quality: The inter-annotator agreement for the English dialogs was 0.87 (Krippendorff's alpha). Manual evaluation of translations indicated high linguistic quality (average fluency score 4.3/5) but noted issues with cultural localization and repetitive dialog structures in some languages.
• Model Performance on Synthetic Data: The baseline multilingual NER model achieved an average lenient F1 score of 89.31 and an exact F1 score of 86.82 on the synthetic dialog test sets. Performance varied by language, with higher scores for German, Spanish, and English, and lower scores for Arabic and Hindi.
• Model Performance on Speech Data: When evaluated on speech-transcribed dialogs, performance decreased moderately (average lenient F1: 85.54; exact F1: 81.81), reflecting the challenges introduced by ASR errors.
• External Validation: Models trained on DialogPII were tested on the real-world CallFriend corpus. While performance dropped compared to in-domain testing, the models achieved encouraging results, particularly under type-agnostic evaluation (macro-average lenient F1 > 83), suggesting the learned representations generalize to spontaneous, real-world conversations.

Significance

The paper claims that DialogPII fills a critical gap in the availability of multilingual resources for conversational de-identification. By providing aligned text and speech resources across 11 languages and diverse scenarios, the dataset facilitates research on:

• Multilingual de-identification and conversational NER.
• Dialog anonymization and privacy-preserving NLP.
• Robustness of sensitive information detection in automatic speech recognition pipelines.

The authors emphasize that while the dataset is synthetic, it allows researchers to develop and test anonymization systems that, once deployed, could make the sharing of genuine conversational data feasible. The release of baseline models and the demonstration of transferability to real-world data (CallFriend) suggest that DialogPII provides a useful foundation for developing robust, multilingual de-identification systems, though the authors note that further work is needed for domain adaptation to naturally occurring conversational data.