PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection

1. Problem Statement

The detection and redaction of Personally Identifiable Information (PII) are critical for regulatory compliance (e.g., GDPR, HIPAA, CCPA). However, the research community lacks a standardized benchmark for PII detection due to three main issues:

• Fragmentation: Existing resources are scattered across domain-specific corpora (general NER, synthetic data, financial filings) with mutually incompatible annotation schemes.
• Lack of Standardization: No single corpus covers the full taxonomy of PII types found in production systems. General NER benchmarks (like CoNLL-2003) miss critical PII categories (financial IDs, credentials), while synthetic datasets use proprietary, non-aligned label schemas.
• Domain Silos: Current systems are "siloed." Rule-based engines (e.g., Microsoft Presidio) handle pattern-matching well but fail on context. General NER models lack PII-specific coverage, and specialized models (e.g., financial NER) fail outside their specific domains. Consequently, there is no rigorous way to compare systems across the full spectrum of PII types.

2. Methodology

The authors developed PIIBench, a unified corpus and evaluation framework through a multi-step construction pipeline:

A. Data Consolidation

Ten publicly available datasets were selected from the HuggingFace Hub, covering diverse domains:

• Financial: finer-139 (SEC filings), gretelai-finance.
• Synthetic PII: ai4privacy series, isotonic-pii-200k, nvidia-nemotron.
• General NER: few-nerd, multinerd, wikiann, conll2003.
• Scale: The raw consolidation yielded 2,369,883 annotated sequences and 133.7 million tokens.

B. Normalization Pipeline

To unify heterogeneous sources, the authors implemented a principled normalization process:

1. Format Conversion: Converted various input formats (integer BIO tags, string BIO labels, character-offset spans, XML tags) into a standard token-level BIO (Begin-Inside-Outside) tagging scheme.
2. Label Mapping: Mapped 80+ source-specific label variants to 48 canonical PII entity types.
   • Example: Merged FIRSTNAME, LASTNAME, PREFIX, and GENDER into a single PERSON category.
   • Example: Merged BITCOINADDRESS, ETHEREUMADDRESS, etc., into CRYPTO_ADDRESS.
   • Preservation: Kept FINANCIAL_ENTITY distinct to preserve the granularity of XBRL tags from financial data.
3. Rare Entity Suppression: Entity types with fewer than 500 "B-tag" mentions were collapsed to the O (Outside) label to prevent models from learning from noise. This removed 32 types (mostly markup artifacts like HTML tags or server configs).
4. Stratified Splitting: Applied an 80/10/10 train/validation/test split. Crucially, the split was performed per-source to ensure the test set preserves the distribution of all ten source domains, preventing data leakage and ensuring representative evaluation.
   • Capping: The finer-139 dataset (which originally comprised ~47% of the data) was capped at 150,000 records during splitting to prevent financial data from dominating the training distribution.

3. Key Contributions

1. Unified Corpus: A massive, multi-domain dataset of 2.37 million sequences covering 48 canonical PII types and 3.35 million entity mentions.
2. Normalization Framework: A reproducible pipeline that resolves 80+ conflicting label schemas into a single, consistent BIO taxonomy.
3. Benchmark Evaluation: The first rigorous cross-system comparison of PII detection across a full taxonomy of entity types, establishing a quantitative lower bound for existing approaches.
4. Open Source: Full dataset construction code, normalization maps, and evaluation scripts are publicly available.

4. Results

The authors evaluated eight published systems on a stratified 1% subset of the test set (1,398 records). The systems included rule-based engines, general-purpose NER models, a PII-specific model, and a financial NER specialist.

• Overall Performance: All systems achieved extremely low performance, with the highest F1 score being only 0.1385 (Microsoft Presidio).
• System Breakdown:
  • Microsoft Presidio (Rule-based): Best overall (F1=0.1385) due to regex patterns for structured data (emails, phone numbers), but failed on contextually inferred entities.
  • General NER (spaCy, BERT, SpanMarker): F1 scores ranged from 0.049 to 0.087. They performed reasonably on common types (PERSON, LOC) but failed completely on PII-specific types (credentials, financial IDs).
  • PII-Specific (Piiranha): Surprisingly low performance (F1=0.046) because it was trained only on synthetic data (ai4privacy-400k) and did not generalize to the diverse, real-world distribution of PIIBench.
  • Financial NER (XtremeDistil FiNER): Demonstrated the "domain-silo" problem most starkly. It achieved high Precision (0.699) but near-zero Recall (0.021). It correctly identified financial entities within its domain but produced zero recall for all other PII types.
• Per-Entity Analysis:
  • Financial Entities: The most frequent type in the test set (29.1% of mentions) achieved F1 = 0.0 for all systems except the financial specialist (which ignored everything else).
  • Credentials: Types like USERNAME, PASSWORD, and API_KEY achieved F1 = 0.0 for all general and financial models.
  • Common Entities: Only LOC and PERSON showed moderate performance (F1 ~0.30), as these are covered by standard NER training data.

5. Significance and Conclusion

• Quantifying the Domain-Silo Problem: The results provide empirical evidence that no single existing model can handle the full spectrum of PII. Systems are highly specialized and fail when exposed to out-of-domain data.
• Benchmark Difficulty: PIIBench presents a substantially harder challenge than any single-source dataset. The low baseline F1 scores (<0.14) indicate a significant gap in current PII detection technology.
• Future Direction: The paper calls for the development of multi-source, fine-tuned models capable of generalizing across domains. It also outlines future work, including cross-source generalization experiments, multilingual expansion, and adding medical/legal domains.

In summary, PIIBench serves as a critical infrastructure for the privacy engineering community, moving PII detection research from fragmented, domain-specific evaluations to a unified, rigorous, and comprehensive benchmark.