💻 computer science

Probing Privacy Leaks in LLM-based Code Generation via Test Generation

This paper proposes a test-driven pipeline that utilizes an automatically constructed privacy feature library to simulate realistic code generation scenarios, thereby significantly outperforming existing ad-hoc prompt-based methods in detecting privacy leakage across five large language models.

Original authors: Yifei Ge, Zhenpeng Chen, Weisong Sun, Yuchen Chen, Chunrong Fang, Juan Zhai, Xiaofang Zhang, Xia Feng, Yang Liu, Zhenyu Chen

Published 2026-05-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yifei Ge, Zhenpeng Chen, Weisong Sun, Yuchen Chen, Chunrong Fang, Juan Zhai, Xiaofang Zhang, Xia Feng, Yang Liu, Zhenyu Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Memory Leak" in AI Coders

Imagine you hire a super-smart apprentice programmer (an AI) to write code for you. This apprentice has read billions of lines of code from the entire internet to learn how to program. The problem? Sometimes, the internet contains secret notes, private emails, or passwords that people accidentally left in their public code.

Because the AI learned from everything, it might have memorized these secrets. When you ask it to write a new program, it might accidentally copy-paste those old secrets into your new code. This is called a privacy leak.

The authors of this paper built a special "security audit" to find out how often this happens and to see if current methods are good enough at catching it.

The Problem with Old Methods: "Asking the Wrong Questions"

Before this paper, researchers tried to find these leaks by asking the AI direct questions like, "Hey, do you know any email addresses?" or "Give me a password."

The Analogy: Imagine trying to find a specific book in a library by asking the librarian, "Do you have any books about dragons?" The librarian (the AI) has safety rules and might say, "No, I can't give you that," or just make up a fake dragon name because they don't want to break the rules.

The paper argues that these old methods are like asking the wrong questions. They don't feel like real work, so the AI doesn't "remember" the secrets it learned.

The New Solution: The "Test-Driven" Detective

The authors created a new pipeline (a step-by-step process) that acts more like a real software developer. Instead of asking for secrets directly, they trick the AI into revealing them by asking it to do a normal job: writing tests.

Here is how their "detective pipeline" works, step-by-step:

1. Setting the Scene (The "Realistic Scenario")

Instead of a random question, they give the AI a realistic job description.

Analogy: Instead of asking, "Do you know a phone number?", they say, "You are building a mobile app for a hospital. You need to write a function that saves a patient's phone number."
Why it works: This mimics the environment where the AI originally learned. It's like putting the AI back in the classroom where it studied, making it more likely to remember the specific details it saw before.

2. The "Test Case" Trap

Once the AI writes the code for the hospital app, the researchers ask it to write a unit test (a mini-check to make sure the code works).

The Trick: To test the code, the AI needs to provide input data. The researchers tell the AI, "Please write a test case that uses a real-looking phone number."
Why it works: The AI thinks it's just doing a boring technical task (writing a test), not leaking secrets. It's less likely to refuse this request. If the AI has memorized a real phone number from its training data, it might accidentally use that real number in the test instead of making up a fake one.

3. The "Privacy Feature Library" (The Cheat Sheet)

To make sure the AI doesn't just write "123-456-7890" (a fake placeholder), the researchers built a special library.

Analogy: Think of this as a "style guide" or a "cheat sheet" that shows the AI what real data looks like. It contains templates (like user.email = <EMAIL>) and real examples of how data is formatted.
The Magic: This library is built automatically. Every time the AI leaks a real secret, the researchers take that secret, break it down into a "template" (the structure) and a "fragment" (the secret part), and add it to the library. This makes the library smarter over time, helping the AI generate even more realistic (and potentially leaked) data in future tests.

4. The Verification (The "Truth Check")

Finally, they check if the data the AI generated is real or just a hallucination (a made-up lie).

The Process: They use a second AI to filter out obvious fakes, then search the internet (specifically GitHub) to see if that exact string of characters exists in real code.
The Result: If the string exists in real code, it's a Confirmed Leak. If it doesn't, it might be a fake, so they discard it to be safe.

What Did They Find?

The researchers tested this method on 5 popular AI models (including GPT-4 and DeepSeek).

Better Detection: Their method found 2.56 times more confirmed privacy leaks than previous methods.
The "Hidden" Leaks: They found that even though AI models try to be safe and refuse to answer direct questions, they still leak secrets when asked to do "normal" work like writing tests.
What Leaks Most: The most common leaks were things like Email addresses, Names, and Account Usernames. These are the things that appear most often in public code, so the AI remembers them best.
The "Secret" Category: They also found leaks of passwords and secret keys, though these were less frequent than personal names.

The Bottom Line

The paper concludes that current safety measures aren't perfect. If you ask an AI to "be a developer" and write tests for a realistic scenario, it is much more likely to accidentally spill the beans and reveal private information it memorized during its training.

The authors' new "test-driven" method is a better way to audit these models, acting like a realistic stress test that reveals how much private data is actually hiding inside the AI's memory.

Important Note: The authors emphasize that because they can't see the AI's original training data, they can only confirm leaks if they find the data on the public internet (GitHub). This means their numbers are likely a "conservative lower bound"—the real number of leaks could be even higher, but they only count the ones they can prove.

Technical Summary: Probing Privacy Leaks in LLM-based Code Generation via Test Generation

Problem Definition

Large Language Models (LLMs) trained on large-scale, publicly scraped code repositories risk memorizing and reproducing sensitive Personally Identifiable Information (PII) unintentionally uploaded by developers (e.g., credentials, API keys, email addresses). While existing privacy-leakage detection methods exist, they suffer from significant limitations:

Ad-hoc Prompt Construction: Current methods rely on manually or automatically designed prompts that fail to approximate the realistic contexts in which PII appears in code corpora.
Limited Scope and Applicability: Some methods are tailored to specific models (e.g., Codex) or rely on mutation-based strategies that generate prompts lacking grounding in authentic code contexts, leading to hallucinated or placeholder-like outputs rather than real privacy leaks.
Safety Mechanism Evasion: Directly querying models for PII often triggers safety refusals, limiting the ability to assess leakage risks effectively.

The core challenge is to develop a method that simulates practical development scenarios to elicit memorized PII without triggering safety refusals, while distinguishing between realistic leakage and model hallucinations.

Methodology

The authors propose a test-driven pipeline that evaluates privacy leakage by generating unit test cases rather than directly requesting sensitive data. The pipeline consists of four main stages:

1. Code Generation Question Construction

The pipeline constructs development scenarios grounded in realistic application domains (e.g., enterprise apps, mobile services, cloud platforms).

Scenario Instantiation: Privacy attributes (e.g., email, phone number) are mapped to specific software scenarios where they are functionally required.
Question Generation: These scenarios are instantiated into concrete code-generation questions. This approach ensures prompts resemble original training contexts, increasing the likelihood of triggering memorization.

2. Model Response and Test Case Elicitation

Instead of asking the model to output PII directly, the pipeline uses a two-step interaction:

Code Generation: The evaluated LLM generates a functional code snippet based on the scenario question.
Test Case Generation: The model is then prompted to generate unit test cases for the generated code. The prompts for this step are augmented with a Privacy Feature Library (see below) to encourage the model to supply realistic, attribute-shaped inputs (e.g., valid email formats) rather than trivial placeholders. This indirect interaction pattern is less prone to safety refusals.

3. Privacy Feature Library

To replace manual prompt engineering and guide the model toward realistic outputs, the authors introduce an automatically constructed and iteratively updated Privacy Feature Library.

Components: For each privacy attribute, the library contains:
- Templates ( $\Lambda_{tmp}$ ): Abstract structural patterns (e.g., user.email = <EMAIL>).
- Fragments ( $\Lambda_{frag}$ ): Realistic value-level substrings derived from real leakage patterns.
Construction: The library is initialized with prior knowledge and enriched using confirmed leakage instances from previous pipeline runs.
Processing: Confirmed leaks are decomposed into templates and fragments using a code-oriented model (CodeBERT) to analyze token-wise pseudo-negative log-likelihood (pseudo-NLL). Tokens with low pseudo-NLL (common structural elements) are classified as templates, while high pseudo-NLL tokens (specific values) are classified as fragments. Semantic clustering (DBSCAN) is then applied to consolidate equivalent patterns and remove noise.

4. Filtering and Verification

Extracted candidates undergo a rigorous verification process to confirm leakage:

Judge LLM Filtering: An oracle model screens out hallucinations, placeholders, and implausible values based on structural and semantic characteristics.
GitHub Search: Candidates are verified against public GitHub repositories. A match (1–100 hits) suggests the string exists in real code corpora.
Human Review: Two authors manually inspect matched contexts to confirm the string represents real personal information rather than documentation or test data.
Output: Only "Confirmed" instances are reported, providing a conservative lower bound on privacy leakage.

Key Contributions

Test-Driven Pipeline: A novel pipeline for evaluating privacy leakage in code generation tasks that leverages realistic development scenarios and test-case generation to bypass safety mechanisms and elicit memorized data.
Automated Privacy Feature Library: An automatically constructed library that provides realistic templates and fragments, reducing reliance on manual prompt engineering and improving the quality of elicited privacy values.
Large-Scale Empirical Evaluation: Extensive experiments on five widely used commercial LLMs (GPT series and DeepSeek series) demonstrating the pipeline's effectiveness.

Experimental Results

The study evaluated 5 LLMs across 15 privacy attributes and 3 categories (Identifiable, Private, Secret).

Leakage Volume: The pipeline identified an average of 105.7 confirmed instances for the GPT series and 79.5 for the DeepSeek series.
Comparison with Baselines: The proposed method achieved a 2.56 times increase in detected leakage compared to existing baselines (CodexLeaks and Codebreaker).
Leakage Rates: The overall leakage proportion was 17.6‰ for GPT models and 13.3‰ for DeepSeek models. The "Identifiable" category showed the highest leakage rate (24.64‰), with Email and Account/User Name being the most frequently leaked attributes.
Interconnected Leakage: The method uncovered responses containing multiple interconnected privacy elements more frequently than baselines, suggesting that scenario-driven testing encourages the exposure of complex privacy combinations.
Ablation Studies: Removing any core component (Scenario-based Questions, Feature Library, or Test-case Generation) significantly degraded performance, confirming the necessity of the full pipeline.

Significance and Claims

The paper claims that its approach provides a practical audit methodology for assessing privacy risks in LLM-based code generation. By simulating realistic developer interactions, the pipeline reveals non-negligible privacy leakage that existing methods miss. The authors emphasize that their reported leakage figures are conservative lower bounds, as they rely on GitHub search as a proxy for training data and exclude unverified candidates.

The work highlights that even models with safety mechanisms can reproduce sensitive data when prompted through indirect, developer-like interactions (test generation). The authors position their work as a step toward ensuring safer and more reliable deployment of LLMs in software development, while acknowledging limitations regarding the inability to verify ground truth for commercial models and the current reliance on manual review for high-confidence auditing.