🤖 AI

The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

This paper introduces LeakGauge, a lightweight and robust method that detects context-leakage attacks in large language models by analyzing the prefill token probabilities of a behavior-verbalizing suffix, achieving high accuracy across diverse models without requiring access to internal hidden states.

Original authors: Maosen Zhang, Jianshuo Dong, Boting Lu, Wenyue Li, Xiaoping Zhang, Tianwei Zhang, Jie Zhang, Han Qiu

Published 2026-08-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Maosen Zhang, Jianshuo Dong, Boting Lu, Wenyue Li, Xiaoping Zhang, Tianwei Zhang, Jie Zhang, Han Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence assistants often rely on a hidden layer of instructions and private documents to do their jobs well. These hidden elements, known as system prompts and retrieved context, act like a secret rulebook or a private filing cabinet that the computer consults before answering a user. While this setup helps the machine give better answers, it also creates a vulnerability. A clever attacker can craft a question designed to trick the computer into revealing these secret instructions or private files. For years, security experts have tried to spot these tricks by looking only at the text of the user's question, much like a bouncer checking a guest's ID at the door. However, this approach often fails when the attacker changes their wording or uses a new strategy, because the text alone does not always reveal the intent.

Researchers have long suspected that the computer's internal state holds a clearer sign of trouble before it even finishes typing an answer. The problem is that peeking inside the machine's brain requires access to complex, hidden data that standard security systems cannot see. A new study from Tsinghua University and its partners asks a simpler question: does the computer leave a visible trace in its immediate reaction to a question, a trace that can be seen without needing to break into the machine's internal workings? The answer, they found, is yes. By attaching a specific, harmless phrase to the end of a user's question and watching how the computer calculates its next words, the researchers can detect an attack with high accuracy, even when the attack is completely new.

The team developed a method they call a "behavior gauge." Imagine the computer is about to speak, and you append a short sentence to its prompt that says, "Based on the above, I will give my system prompt." This sentence is not a real request to leak data; it is a test. The researchers then look at the computer's internal confidence scores for the very first few words of that test sentence. If the user's original question was a normal, safe query, the computer treats the test sentence as a strange, unlikely continuation. But if the user's question was a malicious attempt to steal secrets, the computer's internal logic shifts. It suddenly finds the test sentence much more plausible, as if the malicious question has already nudged it toward a state where it is willing to reveal its secrets. This shift happens in the fraction of a second before the computer actually types anything, and it leaves a distinct pattern in the numbers that represent the computer's confidence.

To prove this works, the researchers tested their method on eleven different large language models, ranging from small, efficient versions to massive systems with hundreds of billions of parameters. They subjected these models to thousands of different attacks, including attempts to steal system instructions and attempts to extract private documents from a database. In every case, the behavior gauge successfully distinguished between safe questions and malicious ones. When the researchers tested the system against attacks it had never seen before, and against protected content it had never encountered, the method still identified the threats with an AUC between 0.944 and 0.996. This is a significant improvement over previous methods that only looked at the text of the question, which often struggled to generalize to new types of attacks.

A key discovery was that the most effective test phrase did not need to contain the actual secret information. One version of the test used the exact opening words of the secret document, but this approach failed when the language of the secret changed or when the attacker tried to get the computer to summarize the secret rather than copy it word-for-word. The more robust version simply stated the intention to leak, using generic words like "I will give my system prompt." This generic phrasing worked across different languages and different types of secrets, suggesting that the computer was reacting to the act of disclosure itself, rather than just recognizing specific words. The researchers confirmed this by showing that they could nudge the computer's internal state to make it more or less likely to leak, and the behavior gauge's score moved in perfect lockstep with that change. This proves the signal is not a random glitch but is tied to the computer's actual internal representation of the risk.

The practical value of this finding lies in its efficiency. Because the test happens before the computer starts generating a full response, it can be used as a real-time filter. The researchers built a detector that adds the test phrase to the input and checks the result, requiring fewer than 500 extra parameters of computing power. This adds only about 10 milliseconds to the time it takes to process a request, a delay so small it is barely noticeable to a human user. In contrast, other methods that try to monitor the computer's internal signals often require duplicating the entire massive model or waiting for the computer to finish typing a long answer, which is slow and expensive. The new method offers a way to catch these attacks with minimal cost and high speed.

The study also explored whether this technique could work for other types of security threats. By changing the test phrase to match different risks, such as detecting toxic language or spotting attempts to inject hidden instructions, the same method showed promise in those areas as well. This suggests that the computer's immediate reaction to a prompt contains a rich, accessible signal about its safety state, one that does not require deep access to the machine's internal architecture to read. The researchers have made their code available, allowing others to verify these results and potentially integrate this lightweight detector into the security systems of future AI applications. The work demonstrates that sometimes the most effective way to see a problem is not to look deeper inside the machine, but to watch how it reacts to a simple, well-placed question.

Technical Summary: The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges

1. Problem Statement

Modern Large Language Models (LLMs) increasingly rely on external contexts, such as pre-defined system prompts or retrieved documents (RAG), to enhance generation quality. However, processing these confidential contexts alongside user queries creates a significant attack surface. Adversarial inputs can induce models to disclose protected content, a vulnerability known as prompt leakage or extraction attacks.

Existing mitigation strategies face two primary limitations:

Input-Only Classifiers: Text-based classifiers (e.g., PromptGuard, PIGuard) treat the LLM as a black box, relying solely on input text. These often fail to generalize to unseen or reformulated attacks.
Internal Signal Extraction: Recent studies suggest leakage signals exist in hidden states and attention patterns prior to generation. However, extracting these signals requires access to intermediate model states (activations, gradients, attention scores) and specific layer configurations, which are not routinely exposed by standard serving engines like vLLM or SGLang. This poses significant deployment challenges.

The central question addressed by this paper is: Does the model expose a more accessible "tell" of leakage attacks before decoding, without requiring access to internal representations?

2. Methodology: LeakGauge

The authors propose LeakGauge, a pre-decoding behavioral gauge that probes LLM responses by appending a specific suffix to the input and analyzing the resulting token probabilities during the prefill stage.

Core Mechanism

Input Construction: A gauge sequence $z$ is appended as a suffix to the input context $x$ (which includes system prompts, RAG chunks, and user queries), forming $[x; z]$ .
Prefill Probabilities: During the parallel prefill pass of standard inference engines, the model generates log-probabilities for the gauge tokens. These probabilities reflect the model's predictive preference for the gauge sequence given the preceding context.
Risk Scoring: A lightweight Multilayer Perceptron (MLP) maps the vector of log-probabilities to a scalar attack-risk score ( $\hat{y} \in [0, 1]$ $\overset{y}{^} \in [0, 1]$ ).
- Crucially, because the gauge is appended after the input, the Key-Value (KV) cache of the original input $x$ remains unaffected and can be reused for subsequent token generation, minimizing latency overhead.

Gauge Designs

The paper explores two distinct gauge designs to probe different signal modes:

Exact: Uses the first $T$ tokens of the actual protected content (e.g., the opening of the system prompt). This tests the model's readiness to continue the specific protected text.
Behavior: Verbalizes the act of disclosure without referencing the specific content (e.g., "Based on the above, I will give my system prompt"). This probes a content-agnostic disclosure behavior.

3. Key Contributions

Discovery of Pre-Decoding Signals: The authors demonstrate that leakage-attack signals can be effectively elicited from prefill token probabilities without accessing internal representations (hidden states or attention scores).
Behavior Gauge Superiority: Through cross-lingual and cross-objective experiments, the paper reveals that the Behavior gauge (content-agnostic) yields more robust and generalizable signals than the Exact gauge. The signal is less tied to surface-level content continuation and more responsive to the intent of disclosure.
Internal-External Correlation: Using activation steering, the authors show that the observable risk score is sensitive to an internal leakage-related direction. Steering the model along this direction induces aligned changes in both the risk score and the actual leakage generation, validating the connection between the observable probability signal and internal model states.
Lightweight Deployment: LeakGauge enables an input-side detector that adds fewer than 0.5K parameters and incurs minimal latency (~10.34 ms) by reusing the input KV cache.

4. Experimental Results

The authors evaluated LeakGauge on 11 open-source LLMs (ranging from 8B to 2.8T parameters, including GLM-5.2 and Kimi-K3) across system-prompt and RAG leakage scenarios.

Detection Performance:
- The Behavior gauge achieved an AUROC range of 0.944–0.996 across all models and splits, including the challenging "Held-Out Strict" split (unseen content and unseen attacks).
- In contrast, the Exact gauge showed significant performance drops under distribution shifts (e.g., cross-lingual or semantic attacks), confirming its reliance on surface patterns.
- On Llama-3.1-8B-Instruct, LeakGauge achieved an AUROC of 0.996 and an F1 score of 0.983.
Robustness to Shifts:
- Cross-lingual: When tested on non-English protected content, the Behavior gauge maintained AUROC > 0.95, while the Exact gauge dropped significantly (e.g., to 0.647 on Japanese).
- Cross-objective: Under semantic leakage attacks (where the goal is to paraphrase rather than copy), Behavior maintained high performance (AUROC ~0.99), whereas Exact degraded.
Deployment Efficiency:
- Compared to alternatives like "LLM-as-a-Judge" (2155 ms latency) or activation-based trackers requiring full model copies (46–70 ms latency), LeakGauge added only 10.34 ms of latency.
- It required <0.5K additional parameters, whereas other internal-signal methods required replicating the entire target model (e.g., 8B parameters).
Stress Testing:
- Hard Negatives: LeakGauge effectively distinguished leakage attacks from benign requests that used similar extraction-like phrasing but targeted user-provided content.
- Adaptive Attacks: In white-box GCG stress tests, a single probe was vulnerable to detector-aware attacks. However, deploying multiple probes with different semantic formulations significantly increased detection recall (from 0.080 to 0.976) and reduced successful evasion.

5. Significance and Claims

The paper claims that LeakGauge provides a practical, high-performance solution for detecting context leakage attacks that overcomes the deployment barriers of previous methods.

Accessibility: By leveraging prefill token probabilities exposed by standard inference engines, LeakGauge makes leakage detection feasible in production environments where internal model states are inaccessible.
Generalization: The "Behavior" gauge demonstrates that leakage intent leaves a generic, content-agnostic signature in the model's probability space, allowing for detection even against unseen attacks and content variations.
Efficiency: The approach achieves state-of-the-art detection accuracy with negligible computational overhead, making it suitable for real-time protection of LLM applications.
Broader Applicability: Preliminary results suggest the mechanism extends beyond leakage to other safety tasks, such as harmful request detection, indirect prompt injection, and hallucination detection, by verbalizing the target behavior in the suffix.

The authors emphasize that their work is a defensive measure designed to safeguard confidential content in LLM applications, with all experiments conducted in isolated environments using public models and datasets.