HybridPII: A Tunable High-Recall Ensemble for Automated PII Detection and Privacy Risk Assessment in Compliance-Critical Applications

Technical Summary: HybridPII – A Tunable High-Recall Ensemble for Automated PII Detection

Problem Statement
The rapid accumulation of textual data across domains creates significant privacy risks due to the presence of Personally Identifiable Information (PII). While regulations like the GDPR and India's DPDP Act mandate strict data protection, existing automated detection methods face a dichotomy:

• Rule-based systems (Regex): Offer high precision for structured data (e.g., emails, credit cards) but suffer from low recall and fail to capture contextual PII (e.g., names, addresses).
• Model-based systems (NER/Transformers): Excel at contextual understanding but often struggle with out-of-vocabulary structured entities and can be computationally expensive.
• The Gap: In compliance-critical applications, the cost of a false negative (missing PII) is significantly higher than a false positive. Current tools, including industry standards like Microsoft's Presidio, often prioritize balanced F1-scores over the high recall necessary to minimize privacy breaches.

Methodology
The authors propose HybridPII, a configurable ensemble model designed to maximize recall while remaining tunable for specific compliance needs. The system architecture consists of four primary modules:

1. Data Generation: A SyntheticPIIGenerator (using the Faker library) created 100 realistic documents across four domains: Business Email, Medical Records, Financial Documents, and Legal Documents. A 30-document subset was used for benchmarking.
2. Regex-Based Detector: A RegexPIIDetector targets structured PII using curated patterns for emails, phone numbers (US/Indian formats), Aadhaar, PAN, credit cards (with Luhn validation), SSN, dates, IPs, and URLs.
3. Ensemble NER Detector: An EnsembleNERDetector utilizes spaCy NER models (en_core_web_sm and en_core_web_lg). Note: The paper acknowledges an installation error with the en_core_web_lg model, suggesting the reported results may effectively rely on the sm model. It employs a voting mechanism where confidence is the proportion of models identifying an entity.
4. Hybrid Fusion Strategy: The core HybridPIIDetector merges results from both detectors:
   • Regex results are prioritized with a weighted confidence (base_confidence * regex_weight, default 0.7).
   • NER results are added if not already detected, weighted by ner_weight (default 0.3).
   • A detect_with_filtering method allows tuning based on a final confidence threshold.
5. Privacy Risk Analyzer: An integrated framework (PrivacyRiskAnalyzer) quantifies document sensitivity by assigning weights to PII types (e.g., SSN=10, EMAIL=5), calculating a normalized risk score (0–100), and flagging compliance relevance (GDPR, DPDP, HIPAA).

Key Contributions

1. High-Recall Hybrid Model: A novel ensemble combining weighted regex and NER, achieving a baseline recall of 0.8324.
2. Comprehensive Benchmarking: Evaluation against four baselines: Regex-Only, NER-Only, a Transformer (BERT) NER model, and the Presidio toolkit.
3. Tunability via Ablation Study: Demonstration that the model's precision-recall trade-off can be optimized. By applying a 0.30 confidence threshold and a 70/30 or 50/50 weight split, the model achieves an optimal F1-score of 0.6519, surpassing the baseline Presidio F1-score of 0.6211.
4. Risk Quantification Framework: An automated system for scoring privacy risk and generating mitigation recommendations aligned with regulatory standards.

Experimental Results

• Recall Superiority: The Hybrid model achieved the highest recall (0.8324) among all tested methods, significantly outperforming Presidio (0.6768) and Regex-Only (0.4618). This confirms its efficacy in minimizing missed PII.
• Precision Trade-off: The high recall resulted in lower baseline precision (0.4626). The system produced 129 False Positives (dominated by NER misclassifying "PAN" as an organization) versus only 23 False Negatives.
• Statistical Significance: Paired t-tests showed the Hybrid model's improvement over NER-Only was highly significant (p ≈ 0, Cohen's d = 1.03). The difference between Hybrid and Regex-Only F1-scores was not statistically significant (p = 0.9760).
• Optimization: The ablation study revealed that tuning the threshold to 0.30 and balancing weights (50/50 or 70/30) yielded the best F1-score (0.6519), making the system competitive with or superior to industry baselines in a balanced configuration.
• Domain Performance: The system successfully assigned higher risk scores to Financial (37.0) and Healthcare (21.0) domains compared to Legal and E-commerce, correctly identifying sensitive PII types like PAN and SSN.

Significance and Claims
The paper claims that HybridPII addresses the critical need for high-recall systems in privacy compliance, where the penalty for missing data outweighs the cost of flagging non-PII. The authors assert that their system is not merely a high-recall specialist but a tunable tool that can be adjusted to achieve state-of-the-art F1-scores (0.6519) surpassing Presidio. Furthermore, the integration of a risk quantification framework elevates the tool from a simple detector to a comprehensive data governance solution.

Limitations and Future Work
The authors modestly acknowledge several limitations:

• Synthetic Data: Reliance on generated data may not fully capture real-world complexities.
• Language: The system is currently English-only.
• Technical Constraints: The en_core_web_lg installation issue may have suppressed potential performance gains.
• Scope: The system detects explicit PII mentions but not inferred or implicit sensitive information.

Future work focuses on reducing false positives via post-processing filters, integrating fine-tuned transformers, adding multilingual support, and validating the system on diverse real-world datasets.