🤖 AI

Addressing Labelled Data Scarcity: Taxonomy-Agnostic Annotation of PII Values in HTTP Traffic using LLMs

This paper proposes and evaluates a multi-stage Large Language Model pipeline that enables flexible, taxonomy-agnostic annotation of Personally Identifiable Information in HTTP traffic, addressing data scarcity through synthetic traffic generation and demonstrating accurate detection across diverse privacy definitions.

Original authors: Thomas Cory, Axel Küpper

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Cory, Axel Küpper

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a privacy inspector for a busy city. Your job is to check every letter, package, and digital message leaving the city to make sure no one is accidentally sending out secret personal information (like home addresses, phone numbers, or medical records). This secret information is called PII (Personally Identifiable Information).

For years, inspectors have used two main tools:

The Rulebook: A rigid list of keywords (like "email" or "SSN"). This works well for simple letters but fails when people write things in weird ways or use code.
The Trained Dog: A machine learning model trained on thousands of examples. It's smart, but it only knows the specific rules it was taught. If the rules change (e.g., a new law defines "health data" differently), the dog needs to be retrained from scratch, which takes a long time and requires a massive supply of real, secret letters to study.

The Problem: Real secret letters are hard to get (because of privacy laws), and the rules for what counts as "secret" are always changing. This leaves inspectors with a shortage of good training data and tools that can't adapt.

The New Solution: The "Super-Translator"

This paper introduces a new approach using Large Language Models (LLMs). Think of an LLM not as a trained dog, but as a super-smart, flexible translator who has read almost everything in the library. You don't need to train this translator on specific rules; you just hand them the rulebook right now (at runtime), and they can instantly understand it.

Here is how the authors built their system, using simple analogies:

1. The Assembly Line (The Pipeline)

Instead of asking the translator to do everything at once, the authors built a three-step assembly line:

Step 1: The Cleaner (Pre-processing): Before the translator sees the message, a machine cleans it up. It unscrambles codes (like turning %20 back into a space) so the translator sees a clear, readable sentence.
Step 2: The Detective & The Scribe (Two-Stage Annotation):
- The Detective: First, the LLM looks at the message and says, "I see a phone number and a name here, but no medical records." It creates a short list of what to look for.
- The Scribe: Then, a second pass focuses only on finding the exact text for those specific items. By narrowing the focus, the Scribe doesn't get confused by other numbers or words.
Step 3: The Editor (Review): A final check looks at the work. If the Scribe missed a number or grabbed the wrong word, the Editor fixes it.

2. The "Fake Letter" Factory (Synthetic Data)

Since the inspectors can't use real people's secret letters to train their tools, the authors built a Factory that makes fake letters.

This factory takes a rulebook (a taxonomy) and asks the LLM to write realistic-looking letters that contain specific secrets (like "John Doe's phone number").
Crucially, the factory also writes down the answers (the ground truth) for every letter it makes.
Why this matters: This solves the "shortage" problem. You can generate thousands of practice letters with perfect answers instantly, without ever touching a real person's private data.

3. The Test Drive (Evaluation)

The authors tested this system with three different "Rulebooks" (Taxonomies):

AI4Privacy: A broad list of common secrets (names, emails, IDs).
mHealth: A specific list for health apps (vital signs, fitness data).
PlayStore: A high-level list used by app stores (e.g., "Financial Data" or "Location").

The Results:

Success: The system worked incredibly well on the first two rulebooks. It could read a message, look at the specific rulebook provided that day, and accurately find the secrets.
Challenge: It struggled a bit more with the "PlayStore" rulebook. The authors explain this is because that rulebook uses very broad categories (like "Financial Data") that are harder to pin down to a specific word in a message than concrete things like "Phone Number."
The "Editor" Step: Interestingly, the final "Editor" step didn't always make things better. Sometimes it fixed mistakes, but other times it introduced new ones. The authors suggest this might be because the Editor was using the same "brain" as the Scribe, so it made the same kinds of mistakes.

The Bottom Line

This paper proves that you don't need to retrain a machine every time privacy rules change. Instead, you can use a flexible AI that reads the rules on the fly.

For Inspectors: You can swap rulebooks instantly without rebuilding your tools.
For Data Scarcity: You can generate your own practice data (fake letters) to test your tools, so you don't need to steal real people's data.

The authors conclude that while this isn't a perfect replacement for lightweight, fast detectors yet, it is a powerful tool for creating high-quality training data and auditing systems as privacy laws evolve. They suggest the best future path is to use this flexible AI to create the data, and then teach smaller, faster machines how to do the job.

Technical Summary: Taxonomy-Agnostic Annotation of PII Values in HTTP Traffic using LLMs

Problem Statement

Automated privacy audits of web and mobile applications increasingly rely on analyzing outbound HTTP traffic to detect Personally Identifiable Information (PII) leakage. However, existing learning-based detectors face two critical limitations:

Data Scarcity: They depend on high-quality, manually labelled datasets, which are labor-intensive to create and difficult to maintain.
Taxonomy Rigidity: These models are tightly coupled to fixed label taxonomies. As privacy definitions evolve or differ across domains (e.g., GDPR vs. mHealth vs. Play Store policies), retraining is required, and performance often degrades significantly with even minor changes to the taxonomy or data distribution.

The paper posits that Large Language Models (LLMs), as generalist models capable of zero-shot or few-shot task performance without retraining, offer a potential solution to enable taxonomy-agnostic annotation of PII values at runtime.

Methodology

The authors propose a comprehensive framework consisting of two main components: a multi-stage annotation pipeline and a synthetic data generator.

1. Multi-Stage Annotation Pipeline
The pipeline decomposes the complex task of PII detection into specialized sub-problems to improve accuracy and handle the variability of HTTP traffic:

Deterministic Pre-processing: HTTP message bodies are normalized to mitigate encoding variability. This includes URL decoding (RFC 3986), resolving HTML character references, and optional application-layer decoding (e.g., Base64) for substrings matching specific syntactic patterns. Both original and normalized representations are preserved for traceability.
Two-Stage Annotation:
- Label-Level Classification: An LLM predicts the set of PII labels ( $L_m$ ) explicitly present in the message, adhering to a strict "explicit-value" requirement (no inference from empty keys).
- Instance-Level Extraction: A second LLM performs targeted Named Entity Recognition (NER) on the normalized body, conditioned on the predicted labels. This stage focuses on boundary detection and disambiguation using contextual cues (key names, value patterns, structural position).
Post-Annotation Review: A third LLM stage acts as a reviewer, refining the initial annotations to correct false negatives, label mismatches, and boundary errors.
LLM Inference Harness: To ensure robustness, the pipeline employs Retrieval-Augmented Generation (RAG) to inject task-relevant examples (structural and label-specific) into prompts. A deterministic validator layer enforces schema compliance and normalizes label outputs (e.g., mapping "E-mail" to "Email") using cosine similarity against the target taxonomy.

2. Synthetic HTTP Message Generator
To address the scarcity of labelled data without exposing real user traffic, the authors introduce a taxonomy-driven generator:

Scenario Generation: The LLM creates scenario templates (e.g., account creation, checkout) based on a provided taxonomy.
Message Instantiation: The LLM generates concrete HTTP requests (and optionally responses) containing explicit PII values.
Validation: A validation layer ensures structural well-formedness, taxonomy compliance, and that all annotated values appear verbatim in the generated payload.
Coverage Tracking: The system monitors label coverage to ensure diverse representation across the taxonomy.

Key Contributions

The paper outlines three primary contributions:

C1 (Annotation Pipeline): A multi-stage, LLM-based approach for granular PII identification in HTTP payloads that supports runtime switching of label taxonomies without retraining.
C2 (Synthetic Generator): An LLM-based mechanism for generating synthetic HTTP traffic with ground-truth PII annotations, reducing reliance on manual curation for shifting taxonomies.
C3 (Evaluation): A comprehensive evaluation across three distinct taxonomies (AI4Privacy, mHealth, and PlayStore) covering different domains and granularity levels to assess flexibility and performance.

Results

The evaluation utilized three datasets derived from the synthetic generator, split into 80/20 for evaluation and RAG exemplars. Performance was measured using Precision, Recall, and F1 scores at label and instance levels.

Overall Performance: The pipeline achieved strong performance on the AI4Privacy (53 classes) and mHealth (38 classes) taxonomies. Label-level detection was near-ceiling (F1 > 0.98) at the classification stage, and instance-level extraction remained high (F1 > 0.95).
Taxonomy Variance: Performance was lower on the PlayStore taxonomy (38 classes), which uses higher-level, policy-facing categories. This suggests that annotation difficulty correlates with the semantic granularity of the taxonomy; broad categories are harder to map to specific payload values than concrete PII types.
Architectural Impact:
- The two-stage configuration (classification followed by targeted extraction) generally outperformed or matched single-stage annotation on AI4Privacy and mHealth, primarily improving instance-level recall.
- On PlayStore, the two-stage approach improved label detection but did not translate to better instance extraction, indicating that noisy classifier outputs can propagate errors in complex taxonomies.
Review Stage: The review stage provided limited net benefit, occasionally degrading performance. The authors suggest this is because the same model family was used for annotation and review, potentially reproducing shared assumptions rather than providing independent correction.

Significance and Claims

The paper claims that LLMs serve as a promising foundation for flexible, taxonomy-agnostic traffic annotation. Key takeaways include:

Feasibility: LLMs can accurately identify and annotate explicitly transmitted PII values when the taxonomy is provided at runtime, eliminating the need for retraining on new label sets.
Data Generation: Synthetic HTTP traffic with ground-truth annotations is a viable substrate for controlled evaluation and few-shot prompting, mitigating the scarcity of labelled real-world data.
Limitations and Scope: The authors are modest regarding the deployment of this approach as a direct replacement for lightweight production detectors. They note that:
- Performance depends heavily on taxonomy design and payload structure.
- Synthetic data, while useful for evaluation, may not fully capture the distribution of real-world traffic (e.g., obfuscation, malformed payloads).
- The current evaluation is limited to explicitly transmitted values in request bodies, excluding headers, cookies, and encrypted traffic.
Future Path: The authors propose a hybrid approach as the most promising path forward: using LLM-based annotation to create or refine labelled datasets under evolving taxonomies, and subsequently distilling these annotations into smaller, discriminative models for high-throughput deployment.