WebPII: Benchmarking Visual PII Detection for Computer-Use Agents

1. Problem Statement

The rise of Computer-Use Agents (AI systems that interact with Graphical User Interfaces via vision and action) introduces critical privacy risks. Unlike traditional automation that uses structured APIs or HTML, these agents operate on rendered screenshots. This creates two major privacy challenges:

1. Training Data Leakage: Models trained on real web data inevitably memorize Personally Identifiable Information (PII), leading to potential data leakage.
2. Inference Exposure: Cloud-hosted inference exposes user screenshots containing sensitive data (names, addresses, payment details) to third-party servers.

The Gap: Existing PII detection methods are insufficient for this context:

• Text-based systems (e.g., Presidio, NER) rely on extracted text strings, missing PII where sensitivity derives from visual context (e.g., a label next to a value) or failing on partially filled forms.
• Document-focused datasets target fixed-layout ID cards, not dynamic web interfaces.
• Scene text datasets localize text but do not distinguish sensitive content from UI chrome.
• No public benchmark exists specifically for visual PII detection in web interfaces.

2. Methodology: The WEBPII Dataset

The authors introduce WEBPII, a fine-grained synthetic benchmark comprising 44,865 annotated e-commerce UI images. The dataset is designed with three core properties to address the limitations of existing data:

A. Extended PII Taxonomy

Beyond traditional PII (names, emails), WEBPII annotates transaction-level identifiers that enable re-identification but are often overlooked:

• Order IDs, tracking numbers, delivery dates.
• Gift messages (containing sender/recipient names).
• B2B purchase order numbers.
• Derived values (taxes, totals).

B. Anticipatory Detection (Fill State Diversity)

Web forms expose PII incrementally as users type. To enable proactive redaction, WEBPII generates annotations for three progressive states for every form field:

1. Empty: Placeholder text only.
2. Partial: Mid-typing state (e.g., "New M" in a city field).
3. Full: Completed data.
   This allows models to learn to trigger redaction during data entry rather than after completion.

C. Scalable VLM-Based Generation Pipeline

To avoid manual annotation of real PII while ensuring realistic layouts, the authors developed a reproduction pipeline:

1. Source Collection: 408 unique layouts from 10 e-commerce brands (e.g., Amazon, Apple, Walmart) across 19 page types.
2. VLM Reproduction: Using Claude Opus 4.5 and OpenCode, the system generates functional React code that reproduces the screenshots.
   • Constraint 1: All PII/product data must be variable references, not hardcoded.
   • Constraint 2: Sensitive elements must include specific data attributes for extraction.
3. Data Injection: At render time, the system injects diverse data configurations (synthetic PII via Faker, product data via Amazon Berkeley Objects, and derived values).
4. Annotation Extraction: Bounding boxes are extracted programmatically from the DOM using the browser's Range API, ensuring pixel-accurate alignment without manual labeling.
5. Human-in-the-Loop: A custom interface delegates natural language corrections to the LLM to fix layout issues (e.g., viewport overflow) or missing attributes. This pipeline required human review for 39% of layouts.

3. Key Contributions

1. First Visual PII Benchmark: WEBPII is the first dataset combining visual localization, semantic PII categories, web interface targeting, extended identifiers, and anticipatory form-state support.
2. WEBREDACT Model: The authors trained a lightweight object detection model, WEBREDACT, specifically for real-time CPU inference.
   • Classes: Grouped into "Text" (PII fields, descriptions, order info) and "Image" (product photos) to simplify detection and improve reliability.
   • Performance: Achieves 0.753 mAP@50 on cross-company generalization tasks with 20ms latency on mid-range CPUs (Intel i5/AMD Ryzen 5), enabling 30fps redaction.
3. Comprehensive Ablation Studies: The paper validates that:
   • Fill State Diversity: Training on partial fills is essential; models trained only on full forms perform significantly worse on partial inputs.
   • Data Injection: Increasing text variants (PII values) per layout improves generalization more than layout diversity alone.
   • Extended Identifiers: Including transaction-level data is crucial for preventing re-identification.

4. Experimental Results

The authors evaluated WEBREDACT against strong baselines on a TestCross-Company split (training on 11 companies, testing on Amazon):

Method	mAP@50	Latency
OCR + Presidio (Text-based)	0.183	1.3s
LayoutLMv3 + LLM (Text-based)	0.357	2.9s
WEBREDACT (Ours)	0.753	20ms
WEBREDACT-LARGE (Ours)	0.842	312ms

• Superiority: WEBREDACT more than doubles the accuracy of the best text-based baseline (0.753 vs. 0.357).
• Efficiency: Text-based methods require 1.3s–2.9s per image (mostly due to OCR and LLM classification), whereas WEBREDACT operates in real-time (20ms) on consumer CPUs.
• Generalization: The model maintains high performance across different page types and companies, though performance drops slightly when generalizing to entirely new visual styles (e.g., Amazon's unique design system).

5. Significance and Impact

• Privacy-Preserving Agents: WEBPII provides the foundational infrastructure for integrating privacy protection directly into computer-use agents. By enabling on-device, real-time redaction, it prevents sensitive user data from ever leaving the local device or being exposed to cloud inference.
• Beyond Redaction: The dataset and model support broader agent capabilities, such as conditioning agent behavior on field sensitivity (e.g., an agent knowing not to click a "Share" button if it contains PII).
• Scalable Data Synthesis: The VLM-based reproduction pipeline demonstrates a viable path for generating large-scale, annotated datasets for complex UI tasks without the prohibitive cost of manual labeling.
• Open Release: The authors release the dataset, the trained models (WEBREDACT and WEBREDACT-LARGE), and the generation pipeline to accelerate research in privacy-preserving computer use.

In conclusion, WEBPII addresses a critical gap in AI safety by providing the first specialized benchmark for visual PII detection, proving that lightweight, real-time visual models significantly outperform traditional text-extraction pipelines in dynamic web environments.