💬 NLP

Locale-Conditioned Few-Shot Prompting Mitigates Demonstration Regurgitation in On-Device PII Substitution with Small Language Models

This paper proposes an on-device PII substitution pipeline using a 1-bit Small Language Model with locale-conditioned rotating few-shot prompting to prevent demonstration regurgitation, revealing that while this approach generates more natural text than rule-based methods, it ultimately harms downstream Named Entity Recognition performance due to insufficient training data variety.

Original authors: Anuj Sadani, Deepak Kumar

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anuj Sadani, Deepak Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a stack of sensitive documents containing real names, addresses, and dates. You need to share these documents for research or training, but you must hide the real private information.

This paper is about a new, smart way to do that hiding, specifically using small computers (like the ones in your phone or laptop) instead of sending data to the cloud.

Here is the story of their discovery, told through simple analogies.

The Problem: The "Blank Sticker" Approach

Traditionally, when companies hide private info, they use a "blank sticker" method. If a document says "John Smith lives at 123 Main St," the software replaces it with [PERSON] and [ADDRESS].

The Issue: While this keeps secrets safe, it ruins the document's usefulness. If you try to teach a computer to recognize names later, it only learns to spot the word [PERSON], not what a real name looks like. It's like teaching a child to recognize dogs by showing them a picture of a blank square labeled "DOG." They won't know what a real dog looks like.

The Solution: The "Fake ID" Factory

The authors built a system that doesn't just stick a blank label on the data; it creates a realistic fake version of the person or place.

The Goal: Change "John Smith" to "Marcus Chen" and "123 Main St" to "456 Oak Ave." The text still looks and feels natural, but the real secrets are gone.
The Constraint: This has to happen entirely on the device (on your laptop), without sending the data to a big server in the cloud. This is like trying to run a high-end movie studio inside a toaster.

The Cast of Characters

To pull this off, they used three tools working together:

The Detective (Privacy Filter): A small AI that scans the text and points out, "Hey, that's a name!" or "That's a date!"
The Creative Writer (Small Language Model): A tiny, super-efficient AI (called Bonsai-1.7B) that tries to invent a fake name or address that fits the context.
The Rule-Follower (Faker): A standard computer program that generates random fake data for things like emails or phone numbers.

The Big Mistake: The "Parrot" Effect

The researchers hit a major snag. When they asked the tiny AI writer to create fake names, it started acting like a parrot.

If you gave the AI a prompt with three examples (e.g., "Real: Alice -> Fake: Bob"), and then asked it to handle a completely different language (like Chinese or German), the AI would ignore the new input and just spit out "Bob" again. It was copying the examples verbatim, regardless of the actual text.

The Discovery:
They thought this happened because the AI was "too small" or "too compressed" (quantized). They tested this by using a slightly larger, less compressed version of the AI. It did the exact same thing.

The Lesson: The problem wasn't the hardware or the size of the AI; it was the instructions (the prompt). The AI was just pattern-matching the examples it was given.

The Fix: The "Rotating Menu"

To stop the AI from being a parrot, they changed how they gave it examples.

Old Way: Give the AI the same three examples every time.
New Way: They created a "menu" of examples for different languages (English, Chinese, German, etc.). For every new piece of text, they randomly picked three examples from the correct language menu to show the AI.
The Result: The AI stopped copying the wrong examples. It started generating "Li Wei" for Chinese names and "Anna Becker" for German names. It finally understood the assignment.

The Surprising Twist: "Natural" isn't always "Better"

This is the most interesting part of the paper. They wanted to see if their new "Creative Writer" (the AI) was better than the "Rule-Follower" (Faker) for training future computers.

The Expectation: They thought the AI-generated names would be more "natural" and therefore better for training.
The Reality: The "Rule-Follower" (Faker) actually won.
- Why? The AI, even with the fix, tended to pick from a very small list of names it had seen in its examples. It was like a chef who only knows how to cook five specific dishes.
- The "Rule-Follower" (Faker) was random. It could generate thousands of different names.
- The Lesson: When training a computer to recognize private data, variety is more important than naturalness. The computer learns better when it sees a huge variety of fake names, even if they look a bit random, rather than a small set of very realistic ones.

The Bottom Line

On-Device is Possible: You can run sophisticated privacy tools on your own computer without sending data to the cloud.
Prompting Matters: Even tiny AI models can fail if you give them the wrong examples. Changing the examples (locale-conditioned prompting) fixed the "parrot" problem.
Variety > Realism: For training AI to spot private data, a generator that produces many random, slightly different fakes is better than one that produces a few very realistic ones.

The authors released all their code and data so anyone can try this "fake ID factory" themselves.

Technical Summary: Locale-Conditioned Few-Shot Prompting for On-Device PII Substitution

Problem Statement
Existing Personally Identifiable Information (PII) redaction tools (e.g., Microsoft Presidio, OpenAI privacy filter) typically replace detected entities with placeholder tokens (e.g., [PERSON], [EMAIL]). While sufficient for compliance, this approach destroys downstream utility: Named Entity Recognition (NER) models trained on redacted text fail to generalize to natural text, and language model perplexity is dominated by the placeholder tokens rather than the text structure.

Substitution—replacing PII with realistic, fake values of the same type—is a superior alternative but faces three constraints: (1) Consistency: "John Smith" must map to the same fake name throughout a document; (2) Type-Preservation: A Chinese name must not be replaced by a US name; and (3) On-Device Execution: The pipeline must run locally without sending real PII to the cloud.

Methodology
The authors propose a fully on-device pipeline combining three components:

Detection: A 1.5 B parameter Mixture-of-Experts (MoE) token classifier (openai/privacy-filter) detects PII spans using BIOES tagging.
Surrogate Proposal:
- SLM Component: A 1-bit quantized Small Language Model (SLM), specifically Bonsai-1.7B (Q1_0), generates contextual surrogates for high-entropy fields (PERSON, ADDRESS, DATE).
- Rule-Based Component: The faker library handles low-entropy fields (EMAIL, PHONE, URL, etc.).
Integration: An EntityResolver groups detected spans by canonical form and label to ensure consistency. Surrogates are spliced back into the text, preserving original whitespace.

Key Technical Finding: Few-Shot Regurgitation
The paper identifies a critical failure mode in extreme-quantization SLMs. When prompted with a naive, fixed three-shot demonstration template, the 1-bit Bonsai-1.7B (and even a 1.58-bit Ternary-Bonsai-1.7B) ignores the input and verbatim "regurgitates" one of the demonstration outputs.

Diagnosis: This is a prompting failure, not a quantization failure. The model pattern-matches the surface structure of the few-shot examples rather than learning the input-to-output mapping.
Solution: The authors implement Locale-Conditioned Rotating Few-Shot Demonstrations.
- A heuristic classifies input into specific locales (e.g., en, zh, ja) and date formats.
- Each locale has a dedicated pool of 4–8 demonstrations.
- For each input, three demonstrations are sampled deterministically using an MD5 hash of the input string.
- This ensures the same entity receives the same context (caching) while different entities sample different subsets, preventing the model from collapsing to a single fixed output.

Experimental Results
The system was evaluated on a 2,000-document multilingual corpus across six locales (US, India, Germany, Mexico, Japan, China) using three modes: redact (placeholders), faker (random substitution), and hybrid (SLM for names/dates, faker for others).

Naturalness (Perplexity): The hybrid mode significantly outperforms faker across all six locales when evaluated by a multilingual causal LM (XGLM-564M). For example, in Japanese, PPL dropped by 32.3% compared to faker.
Consistency & Length: The hybrid mode achieves perfect consistency (1.000) and superior length preservation compared to faker in four of six locales.
Latency: The hybrid mode is significantly slower (~41.2s per document vs. ~1.6s for faker) due to SLM inference, though caching unique entities mitigates this by ~27%.
Downstream NER Utility (The Negative Finding):
- Training an NER model on redact data yields an F1 of 0.000.
- Training on faker data recovers ~68% of the original F1 (0.656 vs. 0.960 baseline).
- Crucially, training on hybrid data yields a lower F1 (0.346) than faker (0.506) in a matched subset experiment ( $p < 0.001$ ).
- Analysis: While hybrid produces more natural text, it suffers from a "narrower surface distribution." The SLM, even with rotating prompts, tends to copy from the limited demonstration pool (regurgitating specific names like "Jennifer Wong" or "David Kim" repeatedly). In contrast, faker provides a wider variety of i.i.d. random values. The paper concludes that downstream NER training benefits more from surface variety than from naturalness.

Significance and Claims
The paper makes three primary contributions:

Reproducible Pipeline: A fully on-device PII substitution system meeting consistency, type-preservation, and local execution constraints.
Failure Mode Identification: The isolation of "few-shot regurgitation" in 1-bit/1.58-bit SLMs, proving it is a prompting artifact rather than a quantization defect.
Methodological Insight: The demonstration that "naturalness" (measured by PPL) is not a proxy for "downstream utility" (measured by NER F1). The authors report an "honest negative finding": while the SLM-based hybrid approach creates more natural text, the random variety of faker is superior for training downstream NER models.

The authors emphasize that their locale-conditioned prompting strategy successfully eliminates qualitative cross-locale failures (e.g., Chinese names becoming English names) but does not fully resolve the vocabulary narrowing inherent to few-shot prompting in small models. Future work suggests expanding demonstration pools or using mixed-source training corpora to balance naturalness and variety.