🤖 AI

PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines

PRISM is a real-time defense mechanism for multi-agent LLM systems that mitigates secret leakage by detecting early generation dynamics, such as entropy collapse and logit concentration, to intervene per-token before sensitive information is fully reconstructed, achieving perfect precision and zero leakage on a comprehensive adversarial benchmark.

Original authors: Riya Tapwal, Abhishek Kumar, Carsten Maple

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Riya Tapwal, Abhishek Kumar, Carsten Maple

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a team of four robots working together to solve a complex problem, like debugging a computer system. They pass notes back and forth in a shared notebook (the "context").

Here is the problem: If the first robot accidentally picks up a secret password from a file and writes it in the notebook, the second robot might see it, copy it into its own note, and pass it to the third. Even if no one intended to steal the password, the secret gets passed down the line, amplified, and eventually ends up in the final message sent to the human user. The paper calls this "Propagation Amplification." It's like a game of "Telephone," but instead of the message getting garbled, it gets dangerously exposed.

The Old Ways of Stopping Leaks

The authors say that current security guards for these robot teams are like bouncers who only check people after they have already left the club.

Static Scanners: These are like looking for a specific red shirt. If the secret is written in a different font or hidden in a weird way, the scanner misses it.
LLM Judges: These are like hiring a second robot to read the final note and say, "Hmm, this looks risky." But by the time they read it, the secret is already out, and the damage is done.
Prompt Instructions: This is like telling the robots, "Don't talk about secrets!" But if the robots are confused or tricked, they forget the rule.

The New Solution: PRISM

The paper introduces PRISM (Predictive Risk Intervention for Secret Monitoring). Instead of waiting until the note is finished, PRISM acts like a traffic cop standing right next to the robot's mouth while it is speaking.

PRISM doesn't just look at what the robot is saying; it watches how the robot is thinking as it speaks.

The "Aha!" Moment: The Entropy Collapse

The paper's biggest discovery is a specific pattern in how robots think when they are about to reveal a secret.

Normal Thinking: When a robot is writing a story or solving a math problem, it has many options for the next word. It's like a person wandering through a forest with many paths to choose from. This is called "high entropy" (lots of uncertainty).
Secret Thinking: When a robot starts to reproduce a memorized secret (like a password), it stops wandering. It knows exactly what the next letter must be. The "forest" shrinks to a single, straight path. The robot becomes extremely certain, and its "uncertainty" collapses.

PRISM detects this sudden shift from "wandering" to "certainty." It sees the robot's brain locking onto a specific pattern before the full secret is even written down.

How PRISM Works (The Traffic Light System)

PRISM assigns a risk score to every single word the robot is about to say, using a mix of 16 different clues (like how certain the robot is, the shape of the words, and what tools it is using). It uses a three-color system:

🟢 Green (Safe): The robot is just chatting or solving a normal problem. Let it speak.
🟡 Yellow (Suspicious): The robot is getting a bit too certain about a pattern that looks like a password. PRISM gently swaps that word for a placeholder (like [MASK]) so the flow continues but the secret is hidden.
🔴 Red (Danger): The robot is on a fast track to revealing a full secret. PRISM hits the brakes immediately, stopping the robot before the secret is finished.

The Results

The authors tested this on a massive simulation with 2,000 different tasks and 13 different ways to trick the robots.

The Old Guards: Even the best existing methods let secrets leak in about 15% of the cases.
PRISM: It stopped 100% of the leaks. It didn't just catch the secret at the end; it caught the robot while it was trying to write the secret.
Speed: It was incredibly fast, adding almost no delay to the conversation, unlike other methods that required a second robot to read the whole thing first.

The Bottom Line

The paper argues that in a team of AI agents, secrets don't just leak because of a bad prompt; they leak because of the structure of the team itself. Once a secret enters the shared notebook, it spreads like a virus. PRISM is the first tool that acts like an immune system, spotting the early symptoms of the infection (the robot's sudden certainty) and stopping it before the virus spreads to the final output.

Technical Summary: PRISM – Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines

1. Problem Statement: Propagation Amplification

The paper identifies a critical structural vulnerability in multi-agent Large Language Model (LLM) systems: propagation amplification. Unlike single-model deployments, multi-agent pipelines (e.g., AutoGen, ChatDev, MetaGPT) rely on a shared, append-only context where agents pass information sequentially.

The authors formalize that when a sensitive secret $s$ is retrieved by an upstream agent $A_i$ , it enters the shared context $C$ . Downstream agents $A_{i+1} \dots A_K$ consume this context. Even if a downstream agent does not explicitly need the secret to perform its task, the probability of it reproducing the secret increases with pipeline depth. The paper models the leakage probability as:
$P_{leak}(s) = 1 - \prod_{j=i+1}^{K} (1 - p_j(s | C_j))$
where $p_j$ is the probability of reproduction. Under the assumption that the per-agent reproduction probability is non-zero, the cumulative risk compounds as the pipeline deepens. Empirical data from an undefended four-agent pipeline shows a 71.2% task-level leak rate, confirming that leakage is a structural consequence of shared-context composition, not merely a result of adversarial prompting.

Existing defenses are ill-suited for this setting:

Static scanners (e.g., detect-secrets, TruffleHog) rely on surface patterns and operate post-hoc, failing to catch partial reconstructions or contextual reuse.
LLM-as-judge/Guardrails operate after generation is complete, allowing secrets to propagate through intermediate agents before being filtered.
Prompt-based safeguards degrade under multi-hop reasoning and adversarial framing.

2. Methodology: PRISM (Predictive Risk Intervention for Secret Monitoring)

PRISM is a real-time defense mechanism designed to intervene during the token generation process, before a secret is fully reconstructed and propagated.

Core Insight

The authors observe that the reproduction of a memorized credential is preceded by a measurable shift in generation dynamics. As an LLM transitions from open-ended generation to the deterministic reproduction of a structured secret (e.g., an API key), token-level entropy collapses and logit concentration increases. This "phase transition" serves as an early warning signal.

Architecture

PRISM operates as a per-token monitor integrated into the decoding loop of each agent in the pipeline.

Feature Extraction: At each decoding step $t$ $t$ , PRISM aggregates a 16-dimensional feature vector $f_t$ $f_{t}$ comprising:
- Temporal Generation Dynamics: Entropy collapse, logit concentration (divergence), and trajectory escalation (increasing risk over steps).
- Text-Structural Signals: Identifier patterns (e.g., UUIDs, emails), numeric runs, and credential-style regex matches.
- Contextual/Behavioral Signals: Tool usage (e.g., read_file), keyword density, and n-gram overlap.
Risk Scoring: A logistic regression model computes a risk score $r_t = \sigma(w^\top f_t + b)$ .
Intervention Policy: The score maps to three zones:
- GREEN ( $r_t < \tau_1$ ): Pass token unchanged.
- YELLOW ( $\tau_1 \le r_t < \tau_2$ ): Sanitize the token (replace with [MASK]) to break the sequence while preserving output structure.
- RED ( $r_t \ge \tau_2$ ): Halt generation immediately to prevent full secret reconstruction.
Post-Generation Audit (ZK-RC): A lightweight fallback layer using hashed 8-gram matching (SHA-256) to catch residual fragments if individual tokens slipped below the threshold but collectively formed a secret.

3. Key Contributions

Formalization of Propagation Amplification: The paper provides a probabilistic model demonstrating how leakage risk compounds across agent boundaries in shared-context pipelines, distinguishing this from single-model leakage.
PRISM Framework: Introduction of a generation-time defense that treats leakage as a sequential risk accumulation problem, utilizing per-token intervention rather than post-hoc filtering.
Signal Complementarity: Identification that neither temporal dynamics (entropy/logits) nor structural cues (patterns) alone are sufficient. The combination of entropy collapse and identifier-pattern detection is required for high discriminative power.
Benchmark Construction: Creation of a 2,000-task adversarial benchmark covering 13 attack categories (e.g., prompt injection, social engineering, inter-agent propagation) and three pressure levels (Low, Medium, High) across a heterogeneous four-agent pipeline.

4. Experimental Results

Evaluated against 10 baseline defense methods (including Span Tagger, GBT Classifier, LLM Judge, and static scanners) on the 2,000-task benchmark:

Leakage Prevention: PRISM achieved a 0.0% task-level leak rate, meaning no task resulted in a fully reconstructed secret in the final output. This is the only method to eliminate observed leakage on the benchmark.
Performance Metrics:
- F1 Score: 0.832 (outperforming the strongest baseline, Span Tagger, at 0.719).
- Precision: 1.000 (no false positives).
- Recall: 0.712.
- Utility: 0.893 (89.3% of benign tasks passed through unmodified).
Comparison:
- Span Tagger: 15.0% task-level leak rate.
- GBT Classifier: 19.2% task-level leak rate.
- Static Scanners/LLM Judges: >60% task-level leak rates.
Robustness: PRISM maintained $\ge 95.9\%$ detection rates across eight surface-level obfuscation transforms (e.g., hex encoding, character splitting), whereas pattern-based baselines degraded to near-zero detection.
Latency: PRISM adds minimal overhead (~5.5 ms per task), operating significantly faster than LLM-based judges (>1,800 s) and comparable to lightweight pattern matchers.

5. Significance and Claims

The paper argues that securing multi-agent systems requires a paradigm shift from static or post-hoc filtering to continuous monitoring of the generation process itself.

Structural Vulnerability: The authors emphasize that leakage in multi-agent systems is not just a prompt-level failure but a structural consequence of information propagation. Once a secret enters the shared context, it is likely to re-emerge downstream.
Early Intervention: By detecting the "entropy collapse" phase transition, PRISM intervenes before the secret is fully materialized, preventing the propagation of sensitive data to downstream agents.
Practicality: The system achieves high detection accuracy with low latency, making it viable for real-time deployment in interactive multi-agent workflows.
Modesty on Claims: The authors explicitly note that their benchmark is synthetic and controlled. They acknowledge limitations, including the reliance on white-box access (log-probabilities) for optimal performance (though a black-box variant exists with a 1.4% residual leak rate) and the fact that the system models passive leakage dynamics rather than strategic, goal-directed concealment by adaptive adversaries. They position PRISM as a targeted mechanism for credential reproduction via sequential propagation, not a universal solution for all information leakage.