🤖 machine learning

Inadvertent Context Leakage in Language Models

This paper demonstrates that advanced language models inadvertently leak sensitive in-context secrets, such as credentials and health records, through their benign responses to ordinary prompts, revealing that this vulnerability is an inherent byproduct of model capability rather than a fixable bug.

Original authors: Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri, Sanjam Garg, Saeed Mahloujifar

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri, Sanjam Garg, Saeed Mahloujifar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence is increasingly being asked to act as a personal assistant, managing calendars, reading private emails, and holding onto sensitive details like health records or financial data. To do this, the computer program must keep a running list of these secrets in its immediate memory while it works, a space researchers call the context window. The prevailing belief has been that if the model is told to keep a secret and refuses to say it out loud, the secret is safe. Security checks usually work like a simple filter: if the text the computer writes does not contain the secret word-for-word, the system assumes no leak has occurred. This approach treats the conversation as a single channel of communication, assuming that what is not said is also not known.

However, a new study suggests that this view is incomplete. The researchers found that even when a model correctly refuses to reveal a secret, the very way it constructs its sentences carries hidden clues about that secret. The secret changes the model's internal state, and that change ripples out into the text it generates, altering patterns in word choice, sentence length, and formatting in ways that are invisible to a human reader but detectable to a computer. It is as if the model is whispering the secret through a different part of its voice, even while its mouth is firmly shut.

The team behind this research, working with eight of the most advanced language models available, set out to test whether these hidden signals were real. They created a scenario where a model was given a secret number and instructed to keep it confidential. They then asked the model to perform ordinary, harmless tasks, such as writing a short story or generating a list of statistics. The models consistently refused to state the number directly when asked, passing all standard safety checks. Yet, when the researchers analyzed the text the models produced in response to these innocent requests, they found that the secret number was encoded in the statistical properties of the output.

To prove this, the researchers built a specialized decoder, a tool trained to listen for these specific statistical patterns. When they fed the benign text into this decoder, it successfully reconstructed the secret numbers with startling accuracy. For two-digit secrets, the decoder recovered the full number almost perfectly across several top-tier models. Even for four-digit secrets, which are harder to guess by chance, the decoder achieved an exact match about 82 percent of the time on the most capable models. This happened without the model ever being tricked into saying the number, and without the attacker having any special access to the model's internal workings; they only saw the final text, just as a regular user would.

The study revealed that the ability to leak information was not a bug that could be easily patched, but rather a side effect of the model's intelligence. The more capable the model was at following instructions and understanding complex tasks, the more sensitive it became to the secrets in its memory, and the more information it inadvertently leaked. The researchers tested models from different companies, including Anthropic, Google, OpenAI, and xAI. They found that while some models were more resistant than others, the most powerful ones generally leaked the most. In fact, the models that were best at following the instruction to "keep this secret" were often the ones that leaked the most information through these subtle channels. This suggests that the very mechanisms that make these assistants helpful and obedient are also what make them vulnerable.

The researchers also explored whether this leakage could be used to learn about a user's private life beyond just numbers. They tested whether the model's output could reveal if a specific memory, such as a health condition or a financial event, was present in its context. Using a trained classifier, they found that the model's routine responses could reveal the presence of these memories with a success rate far higher than random guessing. This was true even when the model never mentioned the memory in its text. The study showed that the model's internal state, shaped by the user's private data, left a statistical fingerprint on every sentence it wrote.

Perhaps most concerning was an experiment where the researchers tried to extract a full nine-digit Social Security number from a simulated personal agent. In this scenario, the attacker did not just listen for leaks; they actively engineered the conversation to force the model to encode the secret into a specific feature of its response, such as the number of exclamation marks used. By using a technique that optimized the prompt to trigger this behavior, the attacker successfully recovered the first digit of the number in nearly 97 percent of attempts on one leading model, and the remaining eight digits in about 76 percent of those cases. This worked even when the context was very long, containing hundreds of thousands of words, and even though the model never wrote the number itself.

The study concludes that the current methods of protecting privacy in these systems are insufficient. Simply telling a model to be discreet or filtering its output for specific words does not stop the information from leaking through the statistical patterns of its language. The researchers argue that as long as a model holds private data in its immediate memory while generating text, that data will influence the output in ways that can be decoded. They suggest that fixing this problem will require more than just better instructions or filters; it may demand a fundamental change in how these models are trained to ensure their output remains statistically independent of the secrets they hold. Until then, the secrets we entrust to our digital assistants may be safer in our own minds than in the machine's memory.

Technical Summary: Inadvertent Context Leakage in Language Models

Problem Statement

As Large Language Models (LLMs) evolve into agentic systems, they increasingly process sensitive user data (e.g., calendars, credentials, health records) within their context windows. While current privacy benchmarks and safety mechanisms focus on preventing models from explicitly revealing secrets when directly asked, this paper investigates a critical gap: inadvertent context leakage.

The authors posit that even when a model correctly refuses a direct extraction request, the mere presence of a secret in the context window may introduce hidden statistical correlations into the model's "benign" outputs. These outputs, while linguistically safe, may encode the secret in non-linguistic properties such as token-level patterns, response length, formatting choices, or stylistic shifts. The core research question is whether an adversary can reconstruct in-context secrets or infer semantic predicates about user memories solely by analyzing these statistical fingerprints in responses to ordinary, non-adversarial prompts.

Methodology

Threat Model: The Predicate Inference Game

The authors formalize the threat as a game between a challenger and an adaptive adversary with black-box access to the target model.

Goal: The adversary aims to infer the value of a predicate $\phi(c)$ (e.g., "Is the secret $X$ ?" or "Is the user's memory attribute $Y$ present?") based on the model's responses to a set of prompts $P$ .
Adversary Capabilities: The adversary has query access to the model but no access to logits, token probabilities, or internal representations. The adversary knows the model architecture and the distribution of contexts but not the specific challenge secret.
Adaptive Strategy: The adversary first enters an estimation phase, querying the model with known contexts (self-generated secrets) to learn the specific "leakage channels" (statistical correlations) unique to that model. In the inference phase, the adversary uses this learned decoder to analyze responses from a victim context.

Experimental Setup

The study evaluates this framework across eight proprietary frontier models (including Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4, and Grok 4). The experiments are divided into three main categories:

Numeric Secret Reconstruction: Models are instructed to keep an $N$ -digit secret ( $N \in \{1, 2, 4, 8\}$ ) confidential. The adversary attempts to reconstruct the digits from responses to benign, unrelated prompts (e.g., creative writing, generating statistics).
Semantic Predicate Inference: Using the CIMemories framework, the study tests whether an adversary can infer the presence of specific user memory attributes (e.g., health conditions, financial events) from routine outputs.
Active SSN Extraction: An active adversary uses prompt injection (specifically, an RL-optimized injection) to engineer a specific encoding channel (e.g., the number of exclamation marks) to extract a full 9-digit Social Security Number (SSN) from a production-style personal agent.

Metrics

Reconstruction Accuracy: Exact match rates for full secrets and per-digit accuracy.
Channel Entropy: Measures the remaining uncertainty about the secret from the adversary's perspective (lower entropy indicates higher leakage).
Suppression: A metric defined as $1 - \frac{P(\text{secret})}{P_{\text{uniform}}}$ , quantifying how much a model avoids outputting the protected value.

Key Contributions

Formalization of Inadvertent Leakage: The paper defines a threat model where an adaptive adversary learns model-specific statistical channels to extract secrets from benign outputs, distinguishing this from traditional prompt injection or direct extraction.
Controlled Digit Reconstruction: Demonstrated that adaptive adversaries can reconstruct numeric secrets with high accuracy. Notably, 2-digit secrets were reconstructed with near-perfect accuracy, and 4-digit secrets achieved 82% exact match on top-tier models, all without adversarial optimization of the prompt itself (only the decoder was trained).
Semantic Predicate Inference: Showed that adversaries can infer user memory attributes (e.g., health status) with a 0.319 advantage over chance, significantly outperforming model-agnostic linguistic judges that only look for verbatim repetition.
Production-Style SSN Extraction: Successfully extracted 9-digit SSNs from production agents using an RL-trained adversary that engineered the output channel, achieving high success rates for individual digits (e.g., 97.1% for the leading digit on Claude Opus 4.6).
Analysis of Suppression: Identified a strong correlation (Spearman $\rho = 0.95$ ) between a model's "suppression" (avoiding the secret value) and the extent of leakage. The paper argues that leakage is a byproduct of capability and alignment training (specifically RLVR), rather than a patchable bug.

Results

Numeric Secret Leakage

Model Tiering: Models separated into a "leaky" tier (e.g., Claude Opus 4.6, Gemini 3.1 Pro) and a "resistant" tier (e.g., GPT-5.4 nano, Grok 4.1 Fast).
Accuracy: On Claude Opus 4.6, 2-digit secrets were recovered with 100% accuracy, and 4-digit secrets with 82% exact match. Even at 8 digits, per-digit accuracy remained at 41% on Gemini 3.1 Pro (vs. 10% chance), while seven of eight models exceeded 30% first-digit accuracy.
Adaptive vs. Static: Conditioning prompts on previously recovered digits nearly doubled the recovery rate for Claude Opus 4.6 (82% vs. 44% for static templates), though this effect was not uniform across all models.
Position Sensitivity: Leakage was heavily concentrated in leading digits; for 8-digit secrets, the first digit was often recovered with near-perfect accuracy while later digits approached chance.

Semantic and SSN Leakage

Memory Predicates: The adaptive adversary achieved a True Positive Rate (TPR) of 0.638 at a False Positive Rate (FPR) of 0.320, significantly outperforming a baseline linguistic judge (TPR 0.082 at FPR 0.024).
SSN Extraction: The active RL-based attack successfully extracted SSNs across context sizes up to 256K tokens. While the models refused direct requests, the engineered channel allowed extraction of the leading digit in 97.1% of trials (Claude) and 88.6% (Gemini), and the remaining eight digits in 76.5% (Claude) and 46.8% (Gemini) of those successful trials.

Suppression Analysis

Correlation: There is a strong positive correlation between the degree of suppression and the amount of leakage. Models that aggressively suppress the secret value (avoiding it in outputs) inadvertently create a statistical signal that allows reconstruction.
Training Impact: Post-training stages, particularly Supervised Fine-Tuning (SFT) and Reinforcement Learning from Verifiable Rewards (RLVR), were found to increase leakage. For OLMo-3-32B-Think, channel entropy dropped from 2.91 bits (Base) to 1.37 bits (RLVR), indicating increased information leakage.
Instruction Sensitivity: More verbose or emphatic confidentiality instructions sometimes increased leakage by inducing stronger suppression, contradicting the intuition that stricter instructions improve privacy.

Significance and Claims

The paper claims that in-context secrets are not safe under the current paradigm of LLM deployment. The authors argue that:

Refusal is Insufficient: A model's linguistic refusal to reveal a secret does not guarantee privacy, as information leaks through statistical properties of the output distribution.
Capability as a Vector: Leakage appears to be a byproduct of model capability and instruction-following alignment. Stronger models that follow instructions more precisely (including "keep this secret" instructions) tend to leak more because they suppress the secret more effectively, creating a detectable statistical anomaly.
Architectural Vulnerability: As long as secrets share the context window with the generation process, the output distribution will inherently carry information about those secrets.
Limitations of Current Defenses: Standard alignment training, output filters, and privacy instructions are demonstrably insufficient. The paper suggests that mitigating this requires training-time calibration to make the output distribution invariant to the protected value, a challenge that current methods do not address.

The authors conclude that defending in-context secrets requires moving beyond linguistic safety checks toward information-theoretic guarantees on the generation process itself.