Surrogate Substitution Preserves PHI Detectability: A Multi-Detector Equivalence Study

Technical Summary: Surrogate Substitution Preserves PHI Detectability

Problem Statement
Clinical text de-identification typically follows two paths: redaction (replacing Protected Health Information, or PHI, with tags like [NAME]) or structure-preserving substitution (replacing PHI with realistic, same-type surrogates like "Anna S." $\to$ "Maria S."). While redaction is safe, it destroys the fluency and distributional properties required for downstream NLP tasks. Structure-preserving substitution aims to maintain these properties, but it relies on an unstated assumption: that the surrogate carries the same downstream signal as the original value.

If a substitution silently degrades the features that PHI detectors, Named Entity Recognition (NER) models, or clinical parsers rely on, the transformation effectively "launders" PHI into forms tools can no longer detect. This creates a dual risk: a utility problem (downstream models fail) and a privacy problem (a second-pass safety net fails to catch the data). Previous evaluations often conflated coverage (how much PHI is found and replaced) with utility (whether the replacement remains detectable), or relied on null-hypothesis significance testing (NHST) which, with large sample sizes, renders even negligible differences "statistically significant" but operationally meaningless.

Methodology
The authors introduce a rigorous, paired multi-detector evaluation protocol designed to isolate the effect of substitution from other variables.

1. Paired Multi-Detector Design: The study evaluates 11 diverse detectors (spanning rule-based, fine-tuned transformers, and frontier LLMs) across 7 benchmarks (including clinical and general PII text in 7 languages). Each document is scored in two conditions: the original ($d$) and the transformed ($d'$). Because the only variable changing is the substitution, any shift in detection is attributable solely to the surrogate.
2. Decoupling Coverage and Utility: Utility is measured only on the spans the de-identifier actually masks. This prevents a low-coverage tool from appearing effective (by only masking easy cases) or a high-coverage tool from appearing ineffective (due to the noise of unmasked spans).
3. Equivalence Testing (TOST): Instead of NHST, the study uses the Two One-Sided Tests (TOST) procedure. With a sample size of 57,112 paired spans, the authors define an equivalence margin ($\Delta$) of $\pm 2$ percentage points. The goal is to prove the change in detectability lies within this bounded range, rather than simply rejecting a null hypothesis of zero difference.
4. Error Typology: For spans where detection is lost, the authors manually categorize failures into generator defects (malformed surrogates, truncation, loss of salience) versus intrinsic detector limits.
5. Control Conditions:
   • Redaction Floor: Replacing PHI with ***** to establish the baseline loss of detectability.
   • Open-Source Baseline: Using the open-source Faker library as a surrogate generator to determine if results are tool-specific or inherent to well-formed substitution.

Key Results

• Statistical Equivalence: Across 11 detectors and 7 benchmarks, recall on masked spans dropped from 76.1% (original) to 74.9% (transformed). The TOST analysis rejects non-equivalence ($p \approx 3 \times 10^{-9}$), confirming the change is statistically bounded within the $\pm 2$ point margin.
• Ranking Preservation: The relative performance ranking of the 11 detectors remained stable after transformation.
• Source of Residual Loss: The small residual loss (1.2 points) is not due to detectors becoming "worse" at PHI. Instead, it concentrates in specific generator defects:
  • Malformed/Truncated Surrogates: e.g., "Chicago" $\to$ "Illino" or "Ciudad de la Habana" $\to$ "Cuidad de la Havana".
  • Loss of Salience: Replacing a famous entity (e.g., "Cedars-Sinai") with an obscure one (e.g., "Vidant") reduces detection in models relying on pre-training familiarity.
  • X-Masking: Replacing IDs/emails with x-runs (e.g., nachorutor@... $\to$ nxxxxxxxxx@...) breaks realistic token patterns, though this is privacy-positive.
• Generality: The open-source Faker baseline performed at least as well as the commercial tool (Custodian Guardian Layer) on masked spans, and significantly better on difficult non-English benchmarks where the commercial tool produced truncated surrogates. This suggests the preservation of detectability is a property of well-formed substitution, not a specific vendor's algorithm.
• Redaction Comparison: While redaction collapsed recall to near 0–4%, the structure-preserving transform retained 93–100% of the detectability on masked spans (depending on the detector and matching mode).

Significance and Claims
The paper claims to provide a rigorous framework for auditing structure-preserving privacy transforms. Its primary contribution is not the validation of a specific commercial tool, but the establishment of a protocol that decouples coverage from utility and uses equivalence testing to avoid the "large-N trap."

The authors conclude that structure-preserving substitution does not hide well-formed PHI from downstream detection. The effect is bounded within a negligible margin ($\pm 2$ points) and preserves detector rankings across a heterogeneous panel. Consequently, such transforms can be inserted into analytics and detection pipelines without materially degrading downstream performance. The study emphasizes that any remaining detectability loss is attributable to the quality of the surrogate generation (e.g., truncation or garbling) rather than a fundamental flaw in the substitution concept itself. The authors release the evaluation subsets, scoring code, and an interactive dashboard to enable the community to audit any future structure-preserving transform using this protocol.