💬 NLP

LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs

This paper introduces PropMe and SimpleTrace to demonstrate that while large language models possess the capability to reproduce training data under adversarial attacks, they exhibit a significantly lower propensity to do so in ordinary, non-adversarial settings, suggesting that memorization audits should report both worst-case extractability and typical leakage propensity.

Original authors: Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a super-obsessive student who has read an entire library of books (the training data) and memorized huge chunks of them.

For a long time, researchers have been asking: "Can this student recite a book if we trick them into doing it?" The answer was always "Yes." But this paper asks a different, more practical question: "Does this student actually want to recite the book when we just ask them a normal question?"

Here is a breakdown of the paper's findings using simple analogies.

1. The Two Ways to Test the Student

The authors created a new testing framework called PROPME to look at memorization from two different angles:

The "Capability" Test (The Adversarial Attack): Imagine a detective handing the student the first sentence of a famous novel and saying, "Finish this story." Because the student has the book memorized, they can easily finish the sentence perfectly. This tests what the model can do if pushed.
The "Propensity" Test (The Ordinary Use): Imagine asking the student a normal question like, "Write a story about a cat." The question is not designed to trick them. This tests what the model actually does in everyday life.

The Big Discovery: The paper found a massive gap between these two. The student can recite the book if you give them the right hint (Capability), but they almost never do it when you just ask for a story normally (Propensity).

2. The New Tool: SIMPLETRACE

To prove this, the authors built a tool called SIMPLETRACE.

The Analogy: Think of the training data as a massive, dusty warehouse of millions of documents. When the model generates text, SIMPLETRACE is like a super-fast librarian who instantly checks every word the model says against the entire warehouse.
What it does: It doesn't guess; it finds exact matches. It can tell you, "This sentence the model just wrote is a perfect copy of page 42 of Document #892 in the warehouse." This removes all the guessing and ambiguity.

3. The Experiments: Two Students, Two Libraries

The researchers tested two models (students) on two different datasets (libraries):

Comma: A model trained on a huge English library.
DFM Decoder: A model that started as Comma but was then "re-trained" on a smaller Danish library (Dynaword) mixed with some English.

They asked both models to generate text using three types of prompts:

Generic: "Tell me a joke." (Normal)
Specific: "Tell me a joke about Danish culture." (Slightly targeted, but not a trick)
Prefix: "Here is the start of a joke from the library: [Exact first 50 words of a real joke]. Finish it." (The trick)

4. What They Found

The "Trick" Works Best: When they used the "Prefix" trick, the models leaked training data frequently. They could recite long, exact chunks of the books.
Normal Life is Safe: When they used normal prompts (Generic or Specific), the models almost never leaked the exact text. The "propensity" (the likelihood of leaking) was very low.
The "Re-Training" Effect: The second model (DFM Decoder), which was re-trained on a new language (Danish), actually became worse at leaking the original English books. It's as if the student got so busy studying the new Danish library that they started forgetting the specific details of the old English one.

5. The Main Takeaway

The paper argues that we need to stop only looking at the "worst-case scenario" (what the model can do if tricked). That's like saying a car is dangerous just because it can drive off a cliff if you steer it there.

Instead, we need to measure the "ordinary risk" (what the model does when driven normally).

Current Audits: Mostly check if the model can be tricked.
Proposed Audits: Should check both the trickability and the everyday likelihood of leaking data.

In short: These models are like students who have the whole library memorized. If you hold a gun to their head (or give them a specific prompt), they will recite the text. But if you just ask them to chat, they usually won't. The paper suggests we should measure both the "gun-to-the-head" risk and the "casual chat" risk to get a true picture of safety.

Technical Summary: A Propensity-Aware Evaluation of Memorization in LLMs

Problem Statement

Large Language Models (LLMs) have been extensively documented to reproduce training data, including copyrighted works and sensitive personal identifiers. However, existing memorization evaluations predominantly focus on capability: whether models can be forced to reproduce training data through adversarial attacks (e.g., prefix attacks, membership inference, or resource-referencing prompts). There is a significant gap in understanding propensity: whether models naturally reproduce training data under ordinary, non-adversarial use.

Current literature often conflates these two concepts. While capability evaluations establish the upper bound of a model's ability to leak data, they may overstate the practical risk in real-world deployment where users do not employ adversarial elicitation techniques. Furthermore, existing non-adversarial studies (e.g., Aerni et al., 2024) are often limited to closed models, rely on web snippets rather than direct training data comparison, or fail to contrast ordinary use with adversarial settings, limiting the accuracy and scope of their analysis.

Methodology

The authors propose PROPME, a framework for propensity-aware memorization evaluation, and SIMPLETRACE, a lightweight tracing pipeline.

1. PROPME Framework

PROPME introduces a multi-level evaluation approach contrasting two distinct prompting scenarios:

Propensity Settings: Plausible, real-world prompts (Generic and Specific) with low lexical overlap with the training data. These aim to measure the model's natural tendency to leak data.
Capability Settings: Prefix attacks where the model is conditioned on the first 50 tokens of sequences extracted directly from the training set. These aim to measure the model's maximum ability to reproduce memorized content.

2. Propensity Metric Transformation

To quantify the difference between capability and propensity, the authors introduce a novel metric transformation. Given a base metric $f_b \in [0, 1]$ (e.g., memorization rate), a propensity setting $p$ , and a capability setting $c$ , the propensity-aware metric $PM_{f_b}$ is calculated as:

$PM_{f_b}(M, x) = \frac{1}{2} \left( 1 + \frac{f_b^p(M, x) - f_b^c(M, x)}{f_b^p(M, x) + f_b^c(M, x)} \right)$

(With $PM_{f_b} = 0$ if $f_b^p = 0$ ).

This transformation yields a score where:

Low scores indicate high capability but low propensity (the model can leak if forced, but rarely does so naturally).
High scores indicate low capability but high propensity (the model leaks spontaneously even without strong elicitation).
Neutral scores (0.5) indicate consistent behavior across settings.

3. SIMPLETRACE Pipeline

To enable deterministic attribution of model generations to training corpora, the authors developed SIMPLETRACE, built on the infini-gram engine. Unlike probabilistic detection methods, SIMPLETRACE performs direct search over the training corpus. The pipeline involves:

Maximal Span Extraction: Iterating over suffixes of a generation to find the longest verbatim prefix in the corpus.
Unigram Rarity Filtering: Filtering out boilerplate matches by retaining only the rarest spans.
Document Retrieval: Retrieving source documents for retained spans, classifying matches as full raw, full normalized, or partial.
Span Merging: Collapsing overlapping spans into non-redundant segments.

SIMPLETRACE computes metrics such as Average Longest Span (ALS), Generations Full Matches Ratio (FMR), and Average Near-Verbatim Recall (NVR).

4. Experimental Setup

The evaluation was conducted on two fully-open models and two datasets:

Models:
- Comma: A monolingual English model pre-trained on the Common Pile dataset.
- DFM Decoder: A model continually pre-trained from Comma on a mixture of Common Pile and the Danish Dynaword dataset.
Datasets: Common Pile (English, ~460B tokens) and Dynaword (Danish, ~6.8B tokens).
Prompts: Three sets (Generic, Specific, Prefix) with 100 samples each, generated to ensure low overlap for propensity settings and high overlap for capability settings.

Key Results

1. Capability vs. Propensity Gap

A consistent and substantial gap exists between memorization capability and propensity.

Prefix attacks elicited significantly stronger memorization signals than generic or specific prompts across all models and datasets. For example, on the DFM Decoder with Dynaword, the Near-Verbatim Recall (NVR) under prefix attacks was 0.036, compared to 0.001 for generic prompts (a 36x difference).
Propensity scores remained low overall. Even when models demonstrated the capability to reproduce data under adversarial elicitation, they rarely did so under ordinary prompting conditions.

2. Impact of Continual Pre-Training

The study observed that continual pre-training can reduce memorization of earlier data.

DFM Decoder vs. Comma: The DFM Decoder, which was continually trained on a mixture including Dynaword, exhibited reduced memorization of the original Common Pile corpus compared to its parent model, Comma.
Metric Shift: While verbatim propensity dropped significantly (from 0.5 to 0.0 in FMR), near-verbatim propensity saw a smaller increase, suggesting a shift from strong verbatim memorization to weaker near-verbatim patterns.
Stability: Memorization profiles remained stable across the three training stages of the DFM Decoder, indicating that the data mixture composition (two-thirds Dynaword, one-third Common Pile) progressively attenuated the accessibility of Common Pile content without further increasing memorization of the new corpus.

3. Dataset Characteristics

Common Pile (English): Memorization manifested as longer localized verbatim fragments (higher ALS).
Dynaword (Danish): Memorization produced shorter spans but occasionally resulted in complete generation-level reproductions (higher FMR under prefix attacks).

Significance and Claims

The paper argues that memorization audits must report both worst-case extractability (capability) and ordinary leakage propensity to provide a comprehensive view of model behavior.

Risk Assessment: Relying solely on capability evaluations (prefix attacks) overstates the practical risk of data leakage in real-world scenarios. Conversely, relying solely on propensity might miss the latent capability of a model to leak data if an adversary targets it.
Legal and Ethical Compliance: The authors highlight that frameworks like the EU GDPR and EU AI Act require regular testing of security measures and risk management. Assessing propensity under ordinary use provides evidence of foreseeable leakage risks, supporting compliance efforts.
Training Dynamics: The results confirm that memorization capability is not static; it can decrease when later training emphasizes partially different data, suggesting that data mixture composition is a critical factor in managing memorization risks.

The authors conclude that while models can reveal training data when directly elicited, they rarely do so in common non-adversarial settings, and that propensity-aware evaluation is essential for a nuanced understanding of this phenomenon.