MultiGraSCCo: A Multilingual Anonymization Benchmark with Annotations of Personal Identifiers

The paper introduces MultiGraSCCo, a multilingual benchmark containing over 2,500 annotated personal identifiers across ten languages, which was created using culturally adapted machine translation of synthetic data to facilitate the development and evaluation of anonymization systems while bypassing privacy regulations associated with real patient data.

The Gist

Imagine you are a doctor trying to teach a computer how to spot a patient's name, address, or birthday in a medical report so it can hide them before sharing the file with researchers. This is crucial for privacy, but there's a huge problem: hospitals are terrified of sharing real patient data because of strict privacy laws. It's like trying to teach someone to drive using a real Ferrari, but the owner refuses to let you touch the car.

This paper introduces MultiGraSCCo, a clever solution to this "data shortage" problem. Here is how they did it, explained simply:

1. The Problem: The "Empty Classroom"

To build a good privacy guard (an AI that hides names), you need a classroom full of examples. But in the real world, those classrooms are empty because real patient data is locked away.

• The Old Way: Researchers usually only had data in English. If you wanted to build a privacy guard for German, Russian, or Arabic, you had nothing to learn from.
• The Risk: If you just translate English data into other languages, the names and places might sound weird (like translating "John Smith" directly into a German name that doesn't exist). This confuses the AI.

2. The Solution: The "Magic Translator"

The authors created a multilingual playground with fake (synthetic) patient data in 10 different languages (including German, English, Arabic, Russian, Turkish, and more).

Here is their step-by-step recipe:

• Step 1: The Source Material. They started with a German dataset called GraSCCo. It's already fake data (like a script for a medical drama), so no real people were harmed.
• Step 2: The "Hidden Treasure" Hunt. They didn't just look for obvious names (Direct Identifiers). They also hunted for Indirect Identifiers.
  • Analogy: Imagine a detective trying to find a suspect. The name is obvious. But what if the suspect is the only 80-year-old male who plays the violin and lives in a specific small town? Even without a name, that combination reveals who they are. The authors taught the AI to spot these subtle clues (like hobbies, family history, or specific dates) that could accidentally reveal a patient's identity.
• Step 3: The Cultural Chameleon. They used a powerful AI (GPT-4.1) to translate the German text into 9 other languages. But they gave it a special rule: "Don't just translate; adapt!"
  • The Magic: If the German text says a patient lives in "Musterstadt" (a fake German town), the AI doesn't just translate the word. It swaps it for a real-sounding town in the target country (e.g., "Toulouse" for French or "Istanbul" for Turkish). It changes names, dates, and street names to fit the local culture perfectly.
  • Why? This ensures the AI learns to recognize patterns of privacy, not just specific German words.

3. The Quality Check: The "Human Taste Test"

You can't just trust a robot to translate medical data perfectly. The authors hired real doctors and medical students who spoke both German and the target language to grade the translations.

• They asked: "Does this sound like a real medical report in your country?"
• "Did the AI change the names to sound local?"
• The Result: The translations scored very high (around 6.3 out of 7). The doctors confirmed that the AI successfully made the fake data feel "native" to each culture.

4. The Experiment: Training the AI Guards

They used this new 10-language dataset to train AI models to find and hide personal information. They tested three scenarios:

1. Monolingual: Training an AI only on French data to find French secrets. (Works well).
2. Zero-Shot: Training an AI only on German data and asking it to find secrets in Russian without any Russian training. (It struggled a bit, like a German speaker trying to guess Russian grammar).
3. Multilingual: Training the AI on German plus a tiny bit of Russian data.
   • The Big Win: Even adding a tiny amount of local data (just 25% of the available text) made the AI significantly better at spotting secrets in that language. It's like giving a student a few practice problems in their native language after studying a textbook in a foreign language; suddenly, everything clicks.

Why Does This Matter?

Think of MultiGraSCCo as a universal training manual for privacy.

• For Researchers: It gives them a safe, legal way to practice building privacy tools without needing real patient data.
• For Low-Resource Languages: It helps countries with fewer digital resources (like Ukrainian or Persian) catch up in privacy technology, because they can now use this high-quality, culturally adapted data.
• For Everyone: It makes it safer to share medical data for research, which could lead to better treatments and cures, without violating anyone's privacy.

In short: The authors built a "fake but realistic" multilingual medical library, taught an AI to make it sound culturally perfect, and proved that this library helps build better privacy guards for the whole world.

Technical Summary

Here is a detailed technical summary of the paper "MultiGraSCCo: A Multilingual Anonymization Benchmark with Annotations of Personal Identifiers."

1. Problem Statement

Accessing sensitive patient data for machine learning (ML) is hindered by strict privacy regulations (e.g., HIPAA in the US, GDPR in Europe). While de-identification is required for data sharing, existing datasets are predominantly English, and non-English clinical data often lacks annotations for Indirect Personal Identifiers (IPIs).

• The Gap: Current de-identification often focuses on direct identifiers (names, dates) but fails to address IPIs (e.g., socioeconomic history, family details, specific medical facility names) which can lead to re-identification when combined.
• The Bottleneck: Creating high-quality, annotated multilingual datasets for privacy research is difficult due to legal restrictions on real patient data and the scarcity of native-speaking medical annotators for low-resource languages.

2. Methodology

The authors propose MultiGraSCCo, a multilingual benchmark created by extending the German GraSCCo (Graz Synthetic Clinical text Corpus) dataset. The methodology involves three main stages:

A. IPI Annotation Expansion

• Base Dataset: The authors started with GraSCCo, a synthetic German clinical corpus previously annotated for 18 HIPAA-style Protected Health Information (PHI) categories.
• New Schema: They expanded the annotation schema to include Indirect Personal Identifiers (IPIs) based on a framework by Baroud et al. (2025). The schema includes 13 categories:
  • High Frequency: Time, Facility Personnel.
  • New Categories: Ethnicity, Languages/Speech, Sexual Orientation, and "Details about Location" (to narrow the scope of direct identifiers).
  • Other: Socioeconomic/Criminal history, Family, Appearance, Hobbies/Lifestyle.
• Process: Two annotators labeled the German texts using the INCEpTION platform, achieving a micro F1-score of 0.83 and macro F1 of 0.73 in inter-annotator agreement.

B. Annotation-Preserving Machine Translation

To generate data for 9 additional languages (English, French, Italian, Arabic, Persian, Polish, Russian, Ukrainian, Turkish), the authors used GPT-4.1 with a specialized pipeline:

1. Preprocessing: Typos and medical abbreviations in the original German text were corrected and expanded using GPT-4.1 to ensure semantic clarity before translation.
2. Translation Strategy: The model was prompted not just to translate, but to culturally adapt the content.
   • Entity Adaptation: Names of people, cities, hospitals, and institutions were replaced with natural-sounding, random equivalents specific to the target country (e.g., translating a German city name to a realistic Polish city name, not just a literal translation).
   • Tag Preservation: XML tags surrounding PHI and IPI spans (e.g., <NAME_PATIENT>) were strictly preserved to maintain annotation alignment.
   • Cultural Context: Dates, zip codes, and medical titles were adapted to local norms (e.g., adjusting school grade systems or hospital names).

C. Evaluation Framework

• Human Evaluation: Native-speaking medical professionals evaluated the translations on four dimensions: General Quality, Grammar, Tag Structure, and Entity Translation Quality (cultural adaptation).
• Automated Experiments: The authors trained Named Entity Recognition (NER) models to detect PHI and IPIs under three scenarios:
  1. Monolingual: Training and testing on the same language.
  2. Cross-lingual (Zero-shot): Training only on German, testing on target languages.
  3. Multilingual: Training on German + varying proportions (25%–100%) of target language data.

3. Key Contributions

1. MultiGraSCCo Benchmark: A publicly available dataset containing over 2,500 annotations across 10 languages (covering Germanic, Romance, Slavic, Semitic, Turkic, and Indo-Iranian families) and 3 writing systems (Latin, Cyrillic, Arabic).
2. IPI Schema: A comprehensive annotation layer for indirect identifiers, addressing the limitations of standard HIPAA-style de-identification in a European context.
3. Culturally Adaptive MT Pipeline: A novel methodology demonstrating that Large Language Models (LLMs) can effectively translate clinical texts while preserving annotation integrity and adapting entities to local cultural contexts, rather than producing literal or placeholder-heavy translations.
4. Empirical Baselines: Extensive performance benchmarks for de-identification models across monolingual, zero-shot, and multilingual settings.

4. Results

Translation Quality

• Human Evaluation: The translations received high scores, with an average General Quality of 6.34/7 and Entity Translation Quality of 6.39/7.
• Annotation Preservation: The pipeline successfully preserved 99.76% of the original annotation spans across all languages.
• Cultural Adaptation: Evaluators noted that names and locations were culturally appropriate, though some minor errors regarding specific medical terminology or local institutional structures (e.g., school grades) were observed in Arabic, Turkish, and Persian.

Model Performance (De-identification)

• Monolingual Performance: Using mmBERT (multilingual ModernBERT), models achieved strong results.
  • PHI Detection: High Micro F1 scores (0.90–0.95) across all languages.
  • IPI Detection: Lower performance (Micro F1 ~0.60–0.80, Macro F1 ~0.40–0.60), indicating that indirect identifiers are harder to detect due to their context-dependency.
• Cross-lingual (Zero-shot): Performance dropped significantly when training only on German, particularly for IPIs (e.g., Arabic Macro F1 dropped to 0.42). This highlights the difficulty of transferring knowledge for subtle, context-dependent entities without target-language supervision.
• Multilingual Training: Adding even small amounts of target-language data (25%) significantly improved performance, often surpassing the zero-shot baseline and approaching monolingual performance.
  • Key Finding: Multilingual training offers an effective compromise, leveraging high-resource German data while benefiting from limited annotations in low-resource languages.

5. Significance and Impact

• Privacy Research: MultiGraSCCo enables the development and testing of anonymization systems that comply with stricter regulations (like GDPR) by addressing IPIs, not just direct identifiers.
• Low-Resource Languages: The study proves that synthetic data generation via culturally adapted MT is a viable solution for overcoming data scarcity in privacy research for non-English languages.
• Legal Safety: Because the source data is synthetic and the translations use randomized, culturally appropriate names, the benchmark allows institutions to train and validate models without legal barriers associated with real patient data.
• Future Work: The authors note limitations in replicating the specific stylistic nuances of real clinical notes in target languages and call for future work to incorporate real-world data distributions.

In summary, MultiGraSCCo provides a critical resource for advancing privacy-preserving NLP, demonstrating that high-quality, culturally adapted synthetic data can effectively bridge the gap between privacy regulations and the need for multilingual medical AI development.