💻 computer science

Brazilian-PHI: Benchmarking Checksum-Validated Recognizers for CPF, CRM, CNS, CNPJ, RG, CEP, and Phone in Portuguese Clinical Text

This paper introduces Brazilian-PHI, the first benchmark for detecting seven types of Brazilian personal health information in clinical text, demonstrating that custom Presidio recognizers with checksum validation significantly outperform default tools and large language models in accuracy, latency, and compliance with LGPD requirements.

Original authors: Igor Eduardo

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Igor Eduardo

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age of healthcare, a patient's medical record is more than just a story of illness and recovery; it is a dense tapestry of personal details that must be protected by law. In Brazil, a specific regulation known as the General Data Protection Law treats health information as a particularly sensitive category, requiring that any computer system handling these notes must first strip away anything that could identify a specific person. This process, called de-identification, is the gatekeeper that allows artificial intelligence to learn from medical data without violating privacy. However, the task is far from simple. Brazilian medical notes contain a unique set of identification codes—numbers for tax records, medical licenses, health cards, and business registrations—that follow strict, complex formats. Unlike simple names or addresses, many of these codes include mathematical checks, much like a security seal, that prove the number is real and not just a random string of digits. If a computer system misses these codes or mistakes a harmless number for a private one, the consequences can be severe, ranging from massive fines to the loss of patient trust.

A researcher named Igor Eduardo set out to solve a specific gap in this field: there was no standard test to see how well different computer tools could find these seven specific Brazilian identification codes within medical text. While tools existed for English or Spanish, they often failed when faced with the unique structure of Brazilian data. To test this, the researcher created a large collection of 500 fake medical notes. These notes were carefully crafted to look real, containing the seven types of identification codes mixed in with hundreds of other numbers that look similar but are harmless, such as medication dosages or hospital facility codes. The goal was to see which method could find the real private codes without getting confused by the harmless ones. The study compared three different approaches: a standard, out-of-the-box software tool, a powerful artificial intelligence model that learns from vast amounts of text, and a custom-built version of the software that was specifically taught to recognize Brazilian formats and perform the mathematical checks on the numbers.

The results revealed a clear divide between how different technologies handle structured data. The custom-built software, which was programmed to look for the specific shapes of Brazilian codes and verify their mathematical seals, performed perfectly. It found every single instance of the seven identification types in the test notes and made no mistakes, correctly ignoring all the harmless numbers that looked similar. In contrast, the standard, unmodified software failed to find most of these codes because it simply did not know what to look for, achieving a very low success rate. The artificial intelligence model, while impressive in its ability to understand language, struggled with the strict rules of these numbers. It found most of the codes but made a few errors, confusing some harmless facility codes with private identification numbers. Crucially, the AI could not perform the mathematical verification that proves a number is valid; it could only guess based on patterns, which is why it occasionally made mistakes that the custom software avoided entirely.

Perhaps the most striking finding was not just about accuracy, but about speed. The custom software processed each medical note in less than ten microseconds, a timeframe so short it is almost instantaneous. The artificial intelligence model, however, took over eight hundred milliseconds to process the same note. This means the custom tool was roughly eighty thousand times faster than the AI. To put this difference in perspective, if the AI were a person reading a page aloud, the custom tool would be a machine that could read the entire library in the time it takes that person to read a single sentence. This massive difference in speed suggests that for the specific job of scrubbing private numbers from medical records, relying on the slow, expensive AI is impractical for large-scale use. The study concludes that the best approach is a hybrid one: using fast, rule-based tools to catch the strict, mathematical codes, and saving the powerful, flexible AI for finding the more complex, unstructured details like names and addresses. This combination offers the speed and reliability needed to protect patient privacy while still allowing advanced technology to improve healthcare.

Technical Summary: Brazilian-PHI Benchmarking Checksum-Validated Recognizers

Problem Statement

The deployment of artificial intelligence in Brazilian healthcare is constrained by the Lei Geral de Proteção de Dados (LGPD), which classifies health data as sensitive and mandates the removal of personally identifiable information (PII) before processing or storage. While clinical Natural Language Processing (NLP) is growing in Brazil, a critical gap exists: there is no benchmark for detecting the seven PII types most prevalent in Brazilian clinical notes. These include the individual tax register (CPF), regional medical council number (CRM), national health card (CNS), corporate register (CNPJ), civil identity card (RG), postal code (CEP), and phone numbers.

Existing tools, such as the stock Microsoft Presidio framework and Large Language Models (LLMs), are insufficient for this specific task. Stock Presidio lacks built-in recognizers for Brazilian formats, and LLMs cannot perform the mathematical checksum validation required to distinguish valid CPF/CNPJ numbers from random numeric strings that merely resemble them. This limitation leads to potential false positives or missed detections, risking regulatory non-compliance.

Methodology

The author introduces Brazilian-PHI, the first benchmark specifically designed for Brazilian clinical PII detection. The study evaluates three distinct approaches against a synthetic corpus of 500 Portuguese clinical notes containing 670 PII instances and 1,940 distractors (e.g., ICD-10 codes, dates, CNES facility codes).

1. The Corpus

The dataset consists of 500 synthetic notes generated via a template-based system. Crucially, all CPF and CNPJ instances are algorithmically generated to pass mod-11 checksum validation. Ground truth is defined by exact character-offset spans. The inclusion of distractors tests the system's ability to avoid false positives on formats that superficially resemble PII.

2. Evaluated Methods

Custom Presidio Recognizers: A set of seven custom recognizers implemented for Microsoft Presidio. These utilize regular expressions for pattern matching and, critically, include mod-11 checksum validation for CPF and CNPJ. This layer is deployed as part of a production clinical AI system.
Default (Stock) Presidio: The unmodified Microsoft Presidio v2.2.362, which lacks specific Brazilian identifier rules.
Groq Llama 3.3 70B: A zero-shot LLM approach using a structured prompt to extract PII spans in JSON format. Due to API rate limits, this was evaluated on a subset of 100 notes.

3. Evaluation Metrics

Performance was measured using precision, recall, and F1 scores based on exact span matching. Additional metrics included macro/micro F1, checksum validation accuracy, distractor false positive rates, and wall-clock latency. Statistical significance was assessed using McNemar's test.

Key Results

Performance Accuracy

Custom Presidio: Achieved a macro F1 score of 1.000 across all seven PII types. The inclusion of checksum validation ensured zero false positives for CPF and CNPJ, as any pattern match failing the mathematical check was rejected.
Default Presidio: Achieved a macro F1 of 0.270. It failed to detect five of the seven types (CRM, CNS, CEP, RG, CNPJ) entirely due to the absence of specific rules. It showed partial success on CPF (F1 = 0.889) by matching generic tax-ID patterns but lacked validation.
Llama 3.3: Achieved a macro F1 of 0.989 on the 100-note subset. While highly accurate, it produced one false positive on a CNPJ-formatted distractor and could not perform checksum validation.

Latency and Scalability

The performance gap in processing speed is the most significant differentiator:

Custom Presidio: Average latency of 9.9 µs per note.
Llama 3.3: Average latency of 801.2 ms per note.
This represents an 80,000× difference in speed. The author notes that achieving real-time throughput (100 notes/second) with Llama 3.3 would require approximately 80,000 parallel inference slots, rendering it economically and infrastructurally unfeasible as a primary scrubber for production-scale Brazilian healthcare systems.

Statistical Significance

McNemar's test confirmed that the Custom Presidio approach is statistically significantly superior to Default Presidio ( $\chi^2 = 313.003, p < 0.001$ ). The comparison between Custom Presidio and Llama 3.3 was not statistically significant on the limited sample size, but the latency constraints remain the primary differentiator.

Significance and Claims

The paper claims to provide the first benchmark for Brazilian clinical PII detection, addressing a regulatory and technical gap in the field. Its primary contributions are:

Validation of Checksums: The study demonstrates that checksum validation (mod-11) is essential for deterministic PII detection of CPF and CNPJ. It transforms detection from probabilistic to deterministic, eliminating a class of false positives that regex-only and LLM-based methods cannot reject.
Hybrid Architecture Recommendation: The author proposes a "principled division of labour." Rule-based recognizers with checksum validation are superior for structured identifiers (CPF, CNPJ, CRM, etc.) due to their speed and mathematical certainty. LLMs or fine-tuned NER models are better suited for unstructured PII (names, free-form addresses).
Production Viability: The results argue that while LLMs are competitive in accuracy, their latency makes them unsuitable as the primary mechanism for LGPD compliance at scale. Custom, rule-based recognizers are presented as the necessary foundation for production systems.
Open-Source Release: The author has released the synthetic corpus, the seven custom recognizers, and evaluation scripts as open-source resources to facilitate further research and development in Brazilian clinical NLP.

The paper concludes with a modest limitation: the corpus is synthetic, and the CNPJ format is scheduled to change in July 2026 to an alphanumeric format, requiring future updates to the recognizers. However, the current work establishes a robust baseline for the existing numeric formats.