🤖 machine learning

Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models

This paper argues that aggregate membership inference metrics like ROC-AUC obscure the severe, capacity-dependent risk of verbatim training-data extraction from black-box language models, demonstrating that specific documents containing identifiers or code are frequently leaked regardless of blind baselines or deduplication, and proposing a per-document probabilistic audit framework via the "leakit" tool to accurately measure this privacy harm.

Original authors: Victor Maricato

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Victor Maricato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a specific, secret recipe was used to bake a giant, mysterious cake. You can't see the kitchen, and you can't ask the baker for the ingredients list. All you can do is ask the baker to make a few more slices of cake based on a tiny crumb you found earlier. If the baker keeps spitting out the exact same secret ingredient list every time you ask, you might guess that the secret recipe was indeed in their memory. This is the world of "Language Models"—super-smart computer programs that learn by reading massive amounts of text from the internet. They are like those bakers, but instead of cakes, they predict the next word in a sentence.

The big question scientists are asking is: Do these digital bakers remember the specific books they ate? This is called "Membership Inference." If a model remembers a specific document, it might accidentally spit out private details, like a person's email address or phone number, that were hidden inside that document. For a long time, researchers tried to measure this by looking at the "average" performance of their tests. They would say, "Hey, our test is 90% accurate!" But just like a weather forecast that says it's "mostly sunny" but misses a sudden, dangerous storm, these average scores might hide the fact that the model is leaking dangerous secrets for a few unlucky people.

This paper, titled "LEAK IT," takes a fresh look at how we catch these digital leaks. The author, led by Victor Maricato, argues that the old way of measuring privacy is misleading. They found that many "smart" tests claiming to detect if a document was in the training data were actually just guessing based on the style of the text itself, not the model's memory. It's like trying to guess if a cake was made with a secret recipe just by looking at the color of the frosting, when the frosting color is actually just a coincidence.

However, the paper reveals a more dangerous truth: while the "average" test might look harmless, the model is leaking, but only for a very specific, small group of documents. The researcher discovered that if you ask the model to continue a sentence from a piece of code or a document containing a real email address, it will sometimes spit out that exact, private information verbatim. It's as if the baker, when asked to finish a sentence about a specific, rare spice, suddenly shouts out the exact brand and batch number of that spice, even though they wouldn't do that for a common ingredient like flour.

The study shows that this "verbatim extraction" gets worse as the models get bigger. For a model with 6.9 billion parameters, about 16.6% of the documents containing real identifiers (like emails or phone numbers) were leaked in this way. Surprisingly, the author found that simply removing duplicate text from the training data (a common safety step) didn't really stop this from happening. They also showed that this risk isn't spread evenly; it's much higher for computer code than for regular stories or articles.

The main takeaway is that we need to stop looking at the "average" privacy score and start checking for these specific, high-risk leaks. The author released a new tool called "leakit" to help auditors find these exact moments where a model might accidentally reveal a private secret. They suggest that instead of just saying "the model is mostly safe," we should admit that for certain documents, the model is a very effective memory leak, and we need stronger protections, like a special privacy shield called "differential privacy," to fix it.

Technical Summary: LEAK IT – A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models

Problem Statement

The paper addresses the evaluation of privacy risks in Large Language Models (LLMs), specifically focusing on Membership Inference Attacks (MIA) and training-data extraction. Current privacy audits predominantly rely on aggregate metrics, such as ROC-AUC, to determine if a model leaks information about its training set. However, recent work (Das et al., 2025) demonstrates that these aggregate metrics are confounded: model-free "blind" classifiers (e.g., bag-of-words, date detection) can often distinguish training members from non-members based solely on surface text statistics, rendering aggregate AUC scores unreliable indicators of actual model leakage.

Furthermore, while sampling-based extraction is known to occur, the extent to which it affects individual documents versus the aggregate population, and how it behaves under strict black-box constraints (no token-level logits, no white-box access), remains under-characterized. The paper argues that aggregate metrics obscure the "tail risk"—a small subset of highly exposed documents that suffer verbatim extraction, even if the average model performance suggests low leakage.

Methodology

Probabilistic Framework

The author proposes a probabilistic lens for analyzing black-box sampling leakage. Given a prefix $x$ , they treat $N$ independent samples $C_1, \dots, C_N \sim p_\theta(\cdot | x)$ as an empirical estimate of the model's output distribution. Leakage signals are defined as functionals of this distribution:

Mean Overlap: The average overlap between samples and a held-out continuation $y$ (corresponding to SaMIA).
Extreme Value: The maximum overlap (or set-union recall) between any single sample and $y$ .
Self-Concentration: The mutual overlap among the samples themselves, measuring how concentrated the distribution is without referencing $y$ .

Experimental Setup

Benchmarks: The study utilizes WikiMIA (to demonstrate the confounding nature of surface-text separability) and MIMIR (an IID train/test split of The Pile with n-gram deduplication between members and non-members).
Blind Baselines: To isolate true model leakage, the author reproduces model-free blind classifiers (date detection, TF-IDF bag-of-words, rare-n-gram selection) and compares them against sampling functionals.
Models: Experiments are conducted on the Pythia model family (410M to 6.9B parameters) and OLMo-1B.
Extraction Audit: A specific probe set of 500 documents from The Pile train split is constructed, containing real email addresses and telephone numbers. The attack involves sampling continuations from a prefix immediately preceding the identifier.
Controls: To attribute leakage to specific documents rather than generic string generation, a mismatched-prefix control is used. A document is counted as "leaked" only if the identifier is reproduced with the correct prefix but not with a mismatched prefix.

Key Results

1. Aggregate Metrics are Confounded

On the WikiMIA benchmark, a blind bag-of-words classifier achieves an AUC of 0.97, while sampling-based functionals (including gold-continuation recovery) perform near chance (AUC ~0.52). On the MIMIR IID split, where temporal shifts are removed, the blind baseline drops to chance (0.56), but sampling functionals (self-concentration, gold-continuation recovery) fail to significantly exceed the residual blind baseline. The incremental AUC for sampling over the blind baseline includes zero in its 95% confidence interval.

Conclusion: As an aggregate membership classifier, black-box sampling adds no significant signal beyond what surface text already reveals.

2. Verbatim Extraction of a "Tail" of Documents

While aggregate metrics fail, sampling successfully verbatim-extracts training data for a specific tail of documents that blind attacks cannot reach.

PII Leakage: On Pythia-6.9B, 16.6% of 500 probe documents (83 documents) had their specific real identifiers (email/phone) reproduced verbatim, a rate that is attributable to the specific document rather than generic emission. For email addresses specifically, the leakage rate is 21.3%.
Scale Dependence: Document-specific leakage grows monotonically with model capacity (from 5.6% at 410M to 16.6% at 6.9B).
Domain Specificity: Leakage is approximately 3× stronger in code than in prose. While prose leakage is lower (4.0% to 12.1% across scales), it remains statistically positive.
Arbitrary Continuation Recovery: Recovery of arbitrary held-out continuations is largely confined to the GitHub domain (member gap +0.44), while being negligible in prose domains (Wikipedia, ArXiv).

3. Extraction Regime Characteristics

Prefix Length: A 16-token prefix is sufficient to trigger extraction.
Sampling Budget: The required sample budget scales inversely with prefix length.
Decoding Parameters: Temperature and nucleus sampling (top-p) have minor effects on the extraction rate.
Deduplication: Training on a deduplicated corpus (Pythia-2.8B-deduped) did not significantly reduce verbatim PII extraction compared to the standard model (12.8% vs 14.0%, $p=0.52$ ). The author notes that near-deduplication may increase the number of passes over the data, potentially offsetting benefits.

Contributions

The paper makes four primary contributions:

Probabilistic Framework: A functional-estimation view of black-box sampling leakage that unifies mean-overlap, extreme-value, and self-concentration statistics as functionals of the model's output distribution.
Extension of Blind-Baseline Critique: Demonstrating that on IID splits, no sampling functional significantly outperforms a model-free baseline as an aggregate membership classifier.
Tail Risk Characterization: Showing that sampling verbatim-extracts training data (including PII) for a tail of documents missed by aggregate metrics, and characterizing the extraction regime regarding scale, domain, and decoding parameters.
Tool Release: The release of leakit, a black-box extraction-audit tool, and the argument that privacy audits should report per-document extraction rates rather than aggregate membership scores.

Significance and Claims

The paper argues that the current practice of reporting aggregate ROC-AUC for privacy audits is insufficient and potentially misleading. It claims that:

Aggregate AUC hides real harm: It averages away the high-risk tail of documents where verbatim extraction occurs.
Per-document disclosure is the critical metric: The ability to reproduce a specific identifier or continuation for a specific document is the concrete privacy harm that MIA aims to detect, and this is invisible to aggregate metrics.
Mitigation Implications: Since deduplication did not measurably reduce extraction in this suite, the author motivates differential privacy as a necessary mitigation strategy that offers formal guarantees.
Audit Standard: Privacy audits should shift focus from aggregate membership inference to per-document extraction analysis.

The author explicitly states that their results do not claim a general failure of deduplication but highlight that, under the tested conditions, it did not protect individual identifiers, reinforcing the need for stronger, guaranteed mitigations.