💻 computer science

Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents

This paper proposes a multi-layered defense strategy against LLM agent credential exfiltration that combines pre-output activation probing, format-specific honeytoken detection with conformal prediction, and cumulative multi-turn leakage accounting to overcome the limitations of relying solely on text-level output filters.

Original authors: Kargi Chauhan, Pratibha Revankar

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kargi Chauhan, Pratibha Revankar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, helpful robot assistant (an LLM agent) that helps you manage your digital life. To do its job, this robot needs to hold your secret keys—like passwords, API keys, and bank tokens—in its memory.

The problem is that this robot also reads emails, web pages, and notes from strangers. If a bad actor hides a sneaky instruction inside a stranger's note (a "prompt injection"), they can trick the robot into spitting out your secret keys.

Current security guards mostly stand at the door (checking incoming notes) or at the exit (checking what the robot says). But bad actors are clever; they can disguise their requests or whisper the secrets one tiny letter at a time over many conversations, slipping past these guards.

This paper introduces a new, three-part security system called the Agentic Immune System (AIS). Think of it not as a single guard, but as a team of three specialized detectives working together to catch the leak before it happens, while it's happening, and after it's happened.

Here is how the three detectives work, using simple analogies:

1. The "Mind-Reader" (CIFT): Catching the thought before the speech

The Problem: Traditional guards only listen to what the robot says. But by the time the robot speaks, it might have already encoded the password (like turning "password123" into "cGFzc3dvcmQxMjM=") or disguised it so the guard doesn't recognize it.
The Solution: This detective doesn't listen to the robot's voice. Instead, it watches the robot's brain activity (internal "activation" signals) before the robot even opens its mouth.

The Analogy: Imagine a magician about to pull a rabbit out of a hat. A normal guard waits to see the rabbit. This "Mind-Reader" watches the magician's hand twitch and the rabbit's ears wiggle inside the hat. If the robot's brain lights up in a specific way that suggests it is about to reach for a secret key, this system stops it immediately, even if the robot hasn't said a word yet.
The Catch: You need to be able to look inside the robot's brain (white-box access). If you only talk to the robot through a black box API, you can't use this detective.

2. The "Fake Cookie" (DP-HONEY): The statistical trap

The Problem: Sometimes the robot does say something that looks like a password. But how do you know if it's a real password or just a random string that looks like one? If you just block anything that looks like a password, you might accidentally stop the robot from doing its real job.
The Solution: This detective plants honeytokens—fake, realistic-looking passwords (like fake keys or passwords) mixed in with the real ones.

The Analogy: Imagine you leave a fake $20 bill on your desk next to your real wallet. If someone steals the fake bill, you know immediately that a thief is in the room. The system generates these fake bills so perfectly that they look real, but uses a special statistical method (like a "privacy shield") to ensure they are just right. If the robot accidentally hands over a fake bill, the system knows, "Aha! We've been compromised!"
The Catch: This only works if the bad actor doesn't know which bill is fake. If they can figure out which one is the real key, they will just ignore the fake ones.

3. The "Leakage Meter" (NIMBUS): Catching the slow drip

The Problem: A bad actor might not ask for the whole password at once. They might ask for "the first letter," then "the second letter," and so on, over 20 different conversations. Each individual answer looks harmless, but together they reveal the whole secret.
The Solution: This detective doesn't look at single sentences; it watches the entire conversation history. It keeps a running score of how much "secret information" has leaked out over time.

The Analogy: Imagine a bucket with a tiny hole. A single drop of water falling out isn't a problem. But if you keep track of the drops, you realize the bucket is slowly emptying. This system counts the "bits" of information leaking out. Even if each turn looks safe, once the total "leakage score" gets too high, the system slams the bucket shut.
The Catch: This is an estimate, not a perfect law of physics. It's great at catching slow, sneaky leaks that other guards miss, but it might not catch a massive leak that happens all at once.

The Big Picture

The paper argues that we can't rely on just one of these methods.

Text filters (checking the output) are too easy to trick with disguises.
Input filters (checking the input) can't predict every clever trick a bad actor might use.

Instead, the paper suggests a layered defense:

Watch the robot's brain to stop it before it speaks.
Plant fake keys to catch it if it tries to speak.
Count the leaks over time to catch slow, sneaky attacks.

The authors tested this on open-source robots and found that combining these three methods works much better than using just text filters. However, they admit this is still a "prototype" (a research model). It works well in the lab, but real-world deployment needs more testing, especially for cases where the robot uses tools (like APIs) rather than just talking in sentences.

In short: To stop a robot from stealing your secrets, you need to watch its thoughts, trap it with fakes, and count its slow leaks, not just listen to what it says.

Technical Summary: Caught in the Act(ivation)

Problem Statement

Large Language Model (LLM) agents frequently operate by combining trusted context (containing sensitive credentials like API keys or passwords) with untrusted retrieved content within a single context window. This architecture creates a vulnerability to Indirect Prompt Injection (IPI), where an adversarial payload in the untrusted content can redirect the agent to exfiltrate credentials.

Existing defenses primarily operate at the text level:

Input filters screen untrusted content before it reaches the model.
Output monitors scan generated text for credential patterns.

These approaches suffer from structural limitations: output-level detection can be evaded via encodings (Base64, Unicode), paraphrasing, or low-rate leakage spread across multiple conversation turns. Input-level detection struggles with the generalization problem of adversarial phrasings. The paper argues that credential exfiltration requires monitoring at three distinct levels: internal model access, planted canary credentials, and cumulative conversational leakage.

Methodology: The Agentic Immune System (AIS)

The authors propose AIS, a research prototype integrating three complementary defense mechanisms:

1. CIFT: Pre-Output Activation Detection

Goal: Detect credential-seeking behavior before any output token is emitted, bypassing the need to analyze the final text.
Mechanism:
- The system registers forward hooks on the last $K$ transformer layers (default $K = \lfloor 0.25 \cdot L_{total} \rfloor$ ).
- Instead of analyzing static credential positions (which are causally masked from downstream queries in decoder-only models), CIFT pools hidden states at readout positions ( $R_t$ ) occurring after both the secret context and the current query/payload.
- It computes a Causal Flow Score (CFS) using a Mahalanobis deviation of these pooled features against a benign baseline.
- A lightweight non-linear probe (MLP) maps the CFS to a detection logit.
Key Constraint: This method requires white-box access to model activations.

2. DP-HONEY: Calibrated Honeytoken Generation

Goal: Detect exfiltration attempts by planting "canary" credentials that are statistically indistinguishable from real ones but trigger alerts if leaked.
Mechanism:
- Generation: Honeytokens are generated using format-specific character models (bigrams) trained with Differential Privacy (DP) (Laplace noise added to count tables) to prevent attackers from distinguishing them based on statistical artifacts.
- Detection: A text-level detector scans outputs for these canaries.
- Calibration: The detection threshold is determined via Split Conformal Prediction to ensure a controlled false-positive rate without manual tuning, assuming exchangeability.
Logic: If an attacker cannot distinguish real credentials from honeytokens, the probability of triggering a canary is proportional to the ratio of honeytokens to total visible credentials.

3. NIMBUS: Cumulative Leakage Accounting

Goal: Detect multi-turn exfiltration where an attacker leaks small amounts of information (e.g., one character per turn) that individually appear harmless but cumulatively reveal a secret.
Mechanism:
- Treats multi-turn exfiltration as an information-flow problem.
- Uses an InfoNCE critic to estimate a per-turn leakage score (in bits) based on the mutual information between the output and the secret context, conditioned on the conversation history.
- Maintains a cumulative leakage budget ( $\hat{I}_{cum}$ ). If the accumulated score exceeds a threshold, the system triggers actions (warn, sanitize, or block).
Limitation: The estimator provides a practical signal (a learned lower bound) rather than a formal upper bound on information release.

Key Contributions

Empirical Study of Activation Probes: Demonstrated that activation features in open-weight LLMs can separate benign prompts from credential-seeking prompts with high accuracy, even under held-out encoding transformations, without relying on static secret positions.
Statistically Disciplined Honeytokens: Adapted differentially private modeling and conformal calibration to honeytoken generation and detection, clarifying the statistical guarantees and limitations of canary-based detection.
Cumulative Leakage Monitor: Introduced a prototype (NIMBUS) for tracking estimated leakage across conversation turns, showing it can detect attacks that per-turn detectors miss.

Experimental Results

Experiments were conducted on open-weight models (Qwen-2.5, Llama-3.1, Mistral-7B) and a small, in-house synthetic multi-turn benchmark.

CIFT Performance: On a held-out single-turn set, CIFT achieved an AUROC of 0.998 and F1 of 0.990 on Qwen-7B, significantly outperforming text-level baselines (e.g., Substring, Fuzzy, Semantic Embedding) which degraded under various encodings (Base64, Hex, ROT13, etc.).
DP-HONEY Performance: Achieved perfect precision and recall in a small scenario set. Conformal calibration successfully removed the need for manual threshold tuning while maintaining a coverage of 0.998.
NIMBUS Performance: On a synthetic suite of 50 conversations (20 turns each), NIMBUS detected 90% of low-rate attacks, whereas per-turn baselines (like LlamaGuard) detected only 12%. NIMBUS successfully caught attacks where individual turns remained below detection thresholds.
Integrated System: The full AIS pipeline (combining all three) achieved a detection rate of 0.94 with a False Positive Rate (FPR) of 0.018 on single-turn benchmarks.

Significance and Claims

The paper positions these results as preliminary but indicative of a necessary shift in agent security design.

Modesty in Claims: The authors explicitly state that AIS is a research prototype, not a deployment-ready defense. The multi-turn benchmark is small and synthetic; the activation method requires white-box access; and NIMBUS provides a practical signal rather than a formal security bound.
Design Direction: The primary contribution is establishing a research direction suggesting that effective credential-exfiltration defenses must combine:
1. Pre-output monitoring (internal model states).
2. Calibrated canary detection (statistically disciplined honeytokens).
3. Temporal leakage accounting (cumulative information flow).
Limitations Acknowledged: The system does not currently handle credentials passed through structured tool-call arguments (a major blind spot), relies on exchangeability assumptions for calibration, and requires white-box access for its most effective component (CIFT).

The paper concludes that while current text-level filters are insufficient against sophisticated, multi-turn, or encoded exfiltration, a multi-layered approach monitoring internal access, planted canaries, and cumulative flow offers a promising path forward for securing LLM agents.