💻 computer science

An Agentic Workflow for Detecting Personally Identifiable Information in Crash Narratives

This paper presents a locally deployable, agentic workflow that combines rule-based extraction, fine-tuned large language models, and an evidence-based verification step to achieve robust and privacy-preserving detection of personally identifiable information in crash narratives, thereby enabling scalable traffic safety analysis while safeguarding individual privacy.

Original authors: Junyi Ma, Pei Li, Rui Gan, Kai Cheng, Steven T. Parker, Bin Ran

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junyi Ma, Pei Li, Rui Gan, Kai Cheng, Steven T. Parker, Bin Ran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of police accident reports. These reports are like treasure troves for safety experts because they tell the story of what happened—why a crash occurred, who was involved, and what the weather was like. This "story" part is called the crash narrative.

However, there's a big problem: these stories often accidentally include private secrets, like a person's home address, phone number, or license plate. If you want to share these stories with researchers to make roads safer, you have to scrub out those secrets first. This is called PII (Personally Identifiable Information) removal.

Doing this by hand is impossible because there are too many reports. Doing it with old-school computer rules is like trying to find a needle in a haystack using a magnet that only picks up iron; it misses the needles that look like grass.

This paper introduces a new, smart system called an Agentic Workflow to solve this. Think of it not as a single robot, but as a specialized detective team working together in a local office (so no data leaves the building, keeping it safe).

Here is how the team works, broken down into simple steps:

1. The "Rule-Follower" (The Hybrid Extractor - Part A)

First, the team has a member who is great at spotting things that look exactly the same every time.

The Job: Finding phone numbers and email addresses.
The Analogy: Imagine a bouncer at a club who only lets in people wearing a specific red hat. If the text looks like a phone number (e.g., 555-0199), this bouncer instantly tags it. It's fast, strict, and never gets confused because phone numbers always follow a pattern.
The Tool: They use a tool called Presidio for this.

2. The "Context Detective" (The Hybrid Extractor - Part B)

Next, the team has a smarter, more creative detective who can read between the lines.

The Job: Finding names, home addresses, and ID numbers.
The Analogy: This detective is like a seasoned librarian. They know that the phrase "140th Street" could be a person's home (PII) or just the name of the road where the crash happened (not PII). A simple rule can't tell the difference, but this detective reads the whole sentence to understand the story.
The Tool: They use a Fine-Tuned AI (a brain trained specifically on crash reports) to do this.

3. The "Second Opinion" (The Verifier)

Sometimes, even the best detectives make mistakes, especially with tricky things like addresses. So, the team has a Verifier.

The Job: Double-checking the tricky finds.
The Analogy: Imagine the Context Detective says, "I think this is a home address!" The Verifier steps in and says, "Show me the proof." The Verifier looks at the text and asks, "Does the story say this is where the person lives, or just where the car crashed?" If the detective can't prove it's a home, the Verifier crosses it out.
The Magic: This step stops the system from accidentally hiding normal road names just because they look like addresses.

4. The "Crowdsourcing" Trick (Ensemble Learning)

To make sure they don't miss anything, the Context Detective doesn't just look at the story once. They look at it five times, like asking five different people to spot a hidden object.

The Analogy: If you ask one person to find a specific word in a book, they might miss it. If you ask five people, you are much more likely to catch it. The system combines all five lists to make a "master list" of potential secrets before sending it to the Verifier.

Why is this a big deal?

Privacy First: The whole team works locally (on the organization's own computers). They don't send the sensitive accident stories to the cloud or a big tech company. It's like doing the work in a locked room rather than a public square.
Better Accuracy: Old methods were either too strict (missing real secrets) or too loose (hiding things that weren't secrets). This team balances both. They found 94% of the real secrets (Recall) while keeping mistakes low (Precision).
Scalable: It can process thousands of reports automatically, making it possible to use this valuable data for safety research without breaking privacy laws.

In a nutshell: The authors built a smart, local, multi-step detective squad. One part uses strict rules for obvious secrets, another uses a smart AI for tricky context, and a third part double-checks the work to ensure no private info slips through and no innocent words get deleted. This makes our roads safer by letting researchers study crash stories without exposing the people involved.

1. Problem Statement

Crash narratives (unstructured text descriptions in police crash reports) are vital for traffic safety analysis but contain Personally Identifiable Information (PII) such as names, home addresses, phone numbers, and license plate numbers.

The Challenge: Manual PII detection is not scalable for large datasets. Existing automated methods struggle because PII in crash narratives is sparse, inconsistent, and highly context-dependent.
The Specific Difficulty: Many strings that look like PII (e.g., "I-43", "140TH ST") are actually roadway identifiers or crash locations, not personal data. Conversely, residential addresses might be embedded in complex sentences.
Privacy Constraints: Many transportation agencies cannot use cloud-based Large Language Models (LLMs) due to data privacy regulations, necessitating locally deployable solutions.

2. Methodology: The Agentic Workflow

The authors propose a locally deployable, two-layer agentic workflow that orchestrates different tools to handle different types of PII, rather than relying on a single monolithic model.

A. Data and Definitions

Dataset: 2017–2022 crash narratives from the Wisconsin Department of Transportation (approx. 100k–150k records/year). A test set of 500 manually annotated narratives was used.
PII Categories:
- Structured: Phone numbers, Email addresses.
- Context-Dependent: Names, Home addresses, Alphanumeric identifiers (Driver's licenses, SSNs, License plates).
Output Format: Span-tagging (e.g., @@@Name@@@) to preserve exact string extraction.

B. Layer 1: Hybrid Extractor

The system routes PII detection tasks to the most suitable tool based on the entity type:

Rule-Based Detector (Microsoft Presidio):
- Task: Detects Phone Numbers and Email Addresses.
- Rationale: These have stable, deterministic patterns. The authors customized Presidio with stricter US phone patterns to reduce false positives.
Fine-Tuned LLM (Llama 3.1-8B):
- Task: Detects Names, Home Addresses, and Alphanumeric Identifiers.
- Training: The model was fine-tuned using Low-Rank Adaptation (LoRA) on 2,000 manually labeled crash narratives. This domain adaptation was crucial as general-purpose PII datasets failed to capture transportation-specific nuances (e.g., distinguishing roadway IDs from license plates).
- Ensemble Learning: To improve recall without changing model weights, the fine-tuned LLM runs $K$ times (e.g., $K=5$ ), and the union of extracted spans is collected.

C. Layer 2: Agentic Verifier

To address ambiguity in the most difficult categories (Home Addresses and Alphanumeric Identifiers), a second agent acts as a "verifier."

Mechanism: A locally deployed Llama 3.1-8B reviews the candidates generated by the Hybrid Extractor.
Prompting: The verifier is instructed to act as a strict reviewer, providing a verbatim evidence snippet from the original text for every decision (KEEP, DROP, or UNCERTAIN).
Logic:
- It distinguishes between a residential address (PII) and a crash location (non-PII).
- It distinguishes between a license plate (PII) and a roadway ID (e.g., "I-94").
- It enforces an audit trail, ensuring decisions are evidence-based.

3. Key Contributions

Domain-Specific Agentic Workflow: A novel framework that decomposes PII detection into specialized sub-tasks (extraction vs. verification) rather than using a single-pass model.
Hybrid Extraction Strategy: Demonstrates that combining rule-based methods (for structured data) and fine-tuned LLMs (for contextual data) outperforms using either approach alone.
Privacy-Preserving Local Deployment: The entire pipeline runs locally on institutional infrastructure, avoiding external API calls and ensuring compliance with strict data privacy regulations.
Evidence-Based Verification: Introduces a verifier agent that reduces false positives in ambiguous categories by requiring explicit textual evidence for retention.

4. Results

The workflow was evaluated against baselines including Presidio, prompt-based LLMs (Llama 8B, Gemma 27B), and a fine-tuned LLM alone.

Overall Performance:
- Precision: 0.82
- Recall: 0.94
- F1-Score: 0.87
- Accuracy: 0.96
- Comparison: Outperformed all baselines. Notably, the fine-tuned Llama 8B (small model) with the agentic workflow outperformed the prompt-based Gemma 27B (large model), proving the value of domain adaptation and workflow orchestration.
Category-Specific Insights:
- Structured PII (Phone/Email): Presidio handled these perfectly (F1=1.00 for email). The workflow inherited this strength.
- Context-Dependent PII:
  - Names: High precision (0.93) and good recall (0.76).
  - Home Addresses: The verifier significantly improved precision to 1.00 (dropping false positives like crash locations), though recall decreased slightly (0.64) due to stricter filtering.
  - Alphanumeric Identifiers: The ensemble + verifier module improved recall (0.75) by recovering missed identifiers, though precision remained moderate (0.54) due to the inherent difficulty of distinguishing road IDs from personal IDs.
Ablation Study: Removing the Ensemble and Verifier modules resulted in a significant drop in performance for Home Addresses and Alphanumeric Identifiers, confirming their necessity for handling ambiguity.

5. Significance and Impact

Scalability: Provides a practical, automated solution for processing massive crash datasets, enabling broader safety research that was previously limited by manual redaction bottlenecks.
Privacy Compliance: Offers a robust path for agencies to utilize LLMs without violating data sovereignty or privacy laws by keeping processing local.
Methodological Shift: Moves the field from "single-model" PII detection to orchestrated, multi-agent workflows that leverage the specific strengths of rules, fine-tuned models, and verification agents.
Safety Interventions: By making crash narratives more accessible (after PII removal), this work facilitates deeper analysis of crash contributing factors, potentially leading to better traffic safety countermeasures.