💬 NLP

Detecting Sensitive Personal Information in Japanese Pre-Training Corpora for Large Language Models

This study presents the first approach to detecting sensitive personal information, specifically Japan's "special care-required personal information" (SCPI), in Japanese pre-training corpora by constructing a dataset via LLM-based annotation and training a machine learning classifier to ensure privacy compliance.

Original authors: Rei Minamoto, Yusuke Oda, Daisuke Kawahara

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rei Minamoto, Yusuke Oda, Daisuke Kawahara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a giant, super-smart robot brain (a Large Language Model) by feeding it every book, article, and webpage on the internet. This is like pouring a massive bucket of water into a swimming pool to make the water deeper. But here's the problem: that bucket of water might accidentally contain some very private, sensitive items—like someone's medical records, criminal history, or details about their disabilities. If the robot brain memorizes these items, it might accidentally spill them out later, which is a big privacy violation.

In Japan, there is a specific law called the APPI that says certain types of private info need "special care." The researchers call this SCPI (Special Care-required Personal Information). Think of SCPI as the "fragile, breakable glass" in your bucket of water that you absolutely must filter out before the robot drinks it.

Here is what this paper did, explained simply:

1. The Problem: No Map for the Japanese "Glass"

While researchers in English-speaking countries have built tools to find and remove this "glass" (sensitive info) from their data, no one had really made a map for Japanese text yet. Also, because this data is so private, you can't just buy a pre-made list of examples to train a computer on. You have to build your own list from scratch, which is tricky because you have to be very careful not to break the law while doing it.

2. The Solution: A Two-Step "Sieve" System

The team built a new system to find this sensitive info in Japanese text. They used a clever two-step process, like a fishing net with two different mesh sizes:

Step 1: The Big Net (Finding Names): First, they used a computer program to find any text that mentioned a person's name. Why? Because in Japanese web data, most private info is attached to a name. If there's no name, it's usually safe to ignore for now. This narrowed down the massive pile of text to a manageable size.
Step 2: The Fine Net (The Smart Filter): Next, they used a very smart, high-powered AI (a "High-Performance LLM") to read those specific texts and decide: "Is this sensitive?"
- The Catch: The super-smart AI is slow and expensive to run on millions of texts. So, they used it only to create a "Gold Standard" training set.
- The Result: They used this small, perfect set of examples to train a lightweight, fast machine learning model. Think of this as training a speedy, cheap robot dog to sniff out the "glass" so you don't have to use the expensive, slow robot brain for the whole job.

3. The Training Data: What Did They Find?

They built a dataset of about 1,000 examples of "sensitive" Japanese text. They focused on three main categories that were easy to find:

Medical History: (e.g., "He has cancer.")
Crime: (e.g., "He was arrested for theft.")
Disability: (e.g., "She uses a wheelchair.")

They also found some tricky cases, like information about sexual orientation or gender identity, which they labeled "LGBT" for their study, even though the law didn't explicitly list them in the same way.

4. The Results: Speed vs. Accuracy

They tested their new "robot dog" (the fast model) against the "super-smart AI" and other methods.

Accuracy: The fast models were surprisingly good at spotting the topic of the sensitive info (like knowing a text is about "medical history"). However, they sometimes struggled to tell the difference between "someone talking about a disease" (general info) and "someone revealing a specific patient's disease" (SCPI).
Speed: This is where the fast models won big. The super-smart AI would take over a month to scan a huge library of text. The fast models could do it in less than a day. It's the difference between a snail and a race car.

5. The "Confusion" and Future Plans

The researchers noticed their fast models sometimes got confused between "medical history" and "disability" because those topics often appear together (like a story about a disabled person getting medical treatment).

What they concluded:

You can't use the super-smart AI to scan the whole internet; it's too slow.
You can use the fast, cheap models to do a "rough pass" and catch the vast majority of sensitive topics.
If you need 100% perfection, you could use the fast model first to filter out the safe stuff, and then use the super-smart AI only on the small pile of "maybe sensitive" stuff left behind.

Important Note on Ethics:
The researchers were very careful. They didn't publish the sensitive data they found. They only published the tool (the classifier) and the method. They also noted that while their tool helps protect privacy, it could theoretically be misused by bad actors to find private info, so they are being very cautious about how they release their work.

In short: They built the first specialized "metal detector" for Japanese text that finds private, sensitive info. It's fast, cheap, and good enough to clean up the massive data lakes needed to train future AI robots, ensuring those robots don't accidentally leak people's secrets.

Technical Summary: Detecting Sensitive Personal Information in Japanese Pre-Training Corpora

Problem Statement
Large-scale pre-training corpora for Large Language Models (LLMs) are often constructed via web crawling, which risks unintentionally collecting personal information. If unfiltered, LLMs may memorize and leak this data. While privacy regulations like the GDPR (EU) and CPRA (US) define sensitive personal information (SPI), research into detecting such data in Japanese text is limited. In Japan, the Act on the Protection of Personal Information (APPI) defines "Special Care-required Personal Information" (SCPI), which includes attributes such as race, creed, medical history, criminal records, and disability. Unlike general Personally Identifiable Information (PII), SCPI requires special protection to prevent unjust discrimination. A critical gap exists in the literature: there are no publicly available SCPI datasets for Japanese, and no systematic studies on detecting SCPI at the corpus level. Furthermore, existing practical pipelines for Japanese LLMs often rely on rule-based filtering or Naive Bayes classifiers that lack semantic completeness.

Methodology
The study proposes a pipeline to construct an SCPI dataset and train a rapid classifier, as illustrated in Figure 1.

Dataset Construction:
- Source: The authors sampled 2.61 million texts from llm-jp-corpus-v3 (a Japanese web corpus from Common Crawl).
- Candidate Extraction: To reduce the search space, a Named Entity Recognition (NER) model was used to identify texts containing person names, as names are the most frequent form of PII in web corpora.
- Two-Stage Annotation:
  - Stage 1 (Low-Parameter Models): Two open-weight models, Llama 3.1 Swallow 8B and Gemma 2 9B, were used to annotate candidate texts. A prompt defined the 11 APPI SCPI categories plus a "non-SCPI" label. Texts were filtered based on confidence scores and model agreement (e.g., requiring both models to agree).
  - Stage 2 (High-Performance Model): A high-performance model (gpt-oss-120b) verified the filtered subset. Manual correction was applied to handle inappropriate outputs and identify edge cases.
- Label Refinement: During manual review, an additional category, "LGBT" (covering sexual orientation and gender identity), was introduced as it was not well captured by the original APPI labels. The final training dataset focused on three categories with sufficient instances: Medical History, Crime (aggregated from criminal record, victimization, and proceedings), and Disability. The dataset was balanced to include 10% positive examples (SCPI) and 90% negative examples (non-SCPI).
Classification Models:
- The study evaluated both traditional machine learning models and Transformer-based models for efficiency.
- Traditional Models: Support Vector Classification (SVC), LightGBM (LGBM), and a simple Neural Network (NN). These utilized vector representations via Doc2Vec, TF-IDF, and a Japanese embedding model (ruri-v3-310m).
- Transformer Model: modernbert-ja-310m, a BERT-based classifier.
- Stacking: To improve recall, the authors employed model stacking, taking the union of predictions from the best-performing base models (SVC and LGBM).

Key Results

Dataset Quality: The constructed dataset contained 954 positive examples (Medical History: 377, Crime: 261, Disability: 316) and 8,586 negative examples. Inter-annotator agreement (Cohen's $\kappa$ ) was 0.614, indicating substantial agreement, though "crime victimization" showed lower agreement due to boundary cases.
Model Performance:
- Cross-Validation: ModernBERT achieved the highest F1 score (>0.95) but was computationally expensive. Among traditional models, LGBM with TF-IDF and SVC with Ruri embeddings performed best.
- Manual Evaluation on Large Corpus: When applied to 1.06 million texts, the classifiers showed a discrepancy between cross-validation precision and manual detection rates. The models were highly effective at detecting SCPI-related category information (e.g., general discussions of medical care) but struggled to distinguish these from actual SCPI instances (specific personal data). The combined proportion of "SCPI" and "Related" texts matched the high precision seen in cross-validation.
- Speed: Traditional models were 4 to 50 times faster than Transformer models. LGBM with TF-IDF could process the entire 184-million-text corpus in under a day on 8 CPU cores, whereas ModernBERT would require over a month on 8 GPUs.
Stacking and Tuning: Stacking SVC and LGBM improved the F1 score to 0.877, with recall increasing by 4.2% compared to the best single model.

Significance and Claims
This study claims to be the first to systematically explore SCPI detection in Japanese text corpora, addressing a critical gap in the literature regarding Japanese privacy compliance for LLMs.

Practical Utility: The authors demonstrate that lightweight, traditional machine learning models are sufficient for the first stage of filtering large-scale corpora. These models can rapidly identify texts containing SCPI-related category information, enabling scalable preprocessing before more expensive, high-precision filtering (potentially using LLMs) is applied to a reduced subset.
Legal Compliance: The research adheres to APPI exceptions for academic research, utilizing LLM-based annotation and manual processing within a strictly local environment to ensure no data leakage.
Limitations: The authors modestly acknowledge that their approach relies heavily on the presence of person names, potentially missing SCPI identifiable only through other attributes (e.g., addresses). Additionally, the reliance on LLM-based annotation means the classifier cannot detect SCPI that the LLM itself fails to identify. The study also notes that while it identified "LGBT" as a relevant category, it falls outside the strict legal definition of SCPI under current APPI, and performance on categories like race and creed was not evaluated due to data scarcity.

In conclusion, the paper provides a foundational dataset and a high-speed classification pipeline for detecting sensitive personal information in Japanese, emphasizing the trade-off between the semantic depth of LLMs and the computational efficiency required for corpus-scale filtering.