💻 computer science

Defense Against Prompt Inversion Attacks: An Information-Theoretic Approach for LLM Collaborative Inference

This paper proposes an information-theoretic defense framework for collaborative LLM inference that utilizes privacy adapters and low-dimensional information bottlenecks to minimize mutual information between intermediate activations and input prompts, thereby achieving superior privacy-utility-latency tradeoffs against prompt inversion attacks.

Original authors: Sayedeh Leila Noorbakhsh, Hossein Khalili, Nader Sehatbakhsh

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sayedeh Leila Noorbakhsh, Hossein Khalili, Nader Sehatbakhsh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Cloud Kitchen" Problem

Imagine you have a very small, low-powered kitchen (your phone or laptop) that wants to cook a complex meal using a world-class recipe book (a Large Language Model, or LLM). Your kitchen doesn't have enough space or tools to cook the whole meal, so you decide to send the ingredients halfway through the process to a giant, professional "Cloud Kitchen" to finish the cooking.

The Problem:
When you send those half-cooked ingredients (the "intermediate activations") to the Cloud Kitchen, a nosy chef there might be able to look at the state of the food and guess exactly what your original secret recipe was. This is called a Prompt Inversion Attack. Even if you just send a few clues, a smart attacker can reconstruct your private input (like a medical diagnosis or a legal secret) just by looking at the halfway point of the cooking process.

The Old Solutions:
Previous attempts to stop this were like putting a little bit of salt or pepper in your ingredients before sending them (adding noise) or trying to hide them in a smaller box (reducing size). The problem is that these methods are often hit-or-miss. They might ruin the taste of the meal (make the AI's answer worse) without actually hiding the secret ingredients very well.

The New Solution: The "Smart Filter" (Privacy Adapters)

The authors propose a new, smarter way to protect your secrets. They call it an Information-Theoretic Defense.

Think of this as installing a Smart Filter (called a "Privacy Adapter") right before you send your ingredients to the Cloud Kitchen.

The Squeeze (Information Bottleneck):
Imagine your ingredients are a huge, colorful pile of vegetables, spices, and meats. The Smart Filter forces this huge pile through a tiny, narrow straw.
- What gets through? The essential structure of the meal (the "syntax" or grammar). The Cloud Kitchen still gets enough info to finish cooking the sentence correctly.
- What gets left behind? The specific, sensitive details (the "semantics" or secret words). Because the straw is so narrow, the unique, rare ingredients (like a specific drug name or a person's ID) get crushed or lost in the squeeze.
The "No-Residual" Rule:
The paper makes a crucial point: You cannot just add a little noise to the ingredients and hope for the best. If you just add noise on top of the original ingredients, a smart chef can mathematically "subtract" the noise and see the original ingredients anyway.
- The Fix: The authors' filter replaces the ingredients entirely with a compressed version. It doesn't add to the pile; it throws the original pile away and only sends the squeezed, compressed version. This makes it mathematically impossible to reverse-engineer the original secret.

How They Train the Filter

The authors didn't just guess how to build this filter. They used a "training camp" approach:

The Defender (You): Tries to make the filter squeeze the ingredients as hard as possible to hide the secrets.
The Attacker (The Cloud Chef): Tries to look at the squeezed ingredients and guess the original recipe.
The Game: They play a game back and forth. The Defender tries to make the Attacker fail, while the Attacker tries to get better at guessing. The goal is to find the perfect balance where the Cloud Chef can still finish the meal (high utility) but cannot guess your secret ingredients (high privacy).

The "Selective Protection" Surprise

One of the coolest findings is that this filter is naturally selective.

Common words (like "the," "is," "and," or punctuation) are like flour or water. They are common and easy to describe even through a tiny straw. The filter lets these pass through easily so the sentence still makes grammatical sense.
Sensitive words (like "cancer," "SSRI," or "flight number 621") are like rare, exotic spices. They are hard to describe with limited space. When the filter squeezes the data, these rare, sensitive words are the first to get lost.

The Result: The Cloud Kitchen can still say, "The flight was good, the staff was friendly," but it completely loses the specific flight number, the route, or the medical condition. The attacker might know that you wrote a review, but they can't tell you what the review was about.

The Results in Plain English

The authors tested this on real-world scenarios (medical notes, legal documents, airline reviews) using powerful AI models.

Privacy: They reduced the attacker's ability to guess your secrets by 15% to 35% compared to other methods. In some cases, the attacker's success rate dropped from nearly 90% down to about 50% (essentially a coin flip).
Quality: The AI's answers were still very good. The "taste" of the meal wasn't ruined.
Speed: The filter is so lightweight that it only slowed down the process by about 6% to 8%. This is fast enough to use on a phone without making you wait.

Summary

The paper introduces a "Smart Filter" for AI that squeezes your data before sending it to the cloud. It mathematically guarantees that sensitive secrets get crushed while keeping the useful structure intact. It's like sending a letter where the envelope is so small that only the general topic fits through, but the specific names and numbers get left behind, all without slowing down the mail delivery.

Technical Summary: Defense Against Prompt Inversion Attacks via Information-Theoretic Collaborative Inference

Problem Statement
Collaborative edge-cloud inference allows resource-constrained devices to leverage Large Language Models (LLMs) by executing initial layers locally and offloading intermediate activations to a cloud server. However, this architecture exposes sensitive user prompts to prompt inversion attacks, where adversaries reconstruct the original input from shared representations. Existing defenses, such as heuristic noise injection or dimensionality reduction, lack principled theoretical guarantees regarding privacy leakage and often fail to address the fundamental tension between preserving task utility and protecting token identity. Furthermore, prior work has shown that token identity persists through all transformer layers, rendering simple depth-based obfuscation ineffective.

Methodology
The authors propose an information-theoretic defense framework that models privacy leakage as the mutual information (MI) between the input prompt $x$ and the transmitted protected activations $A$ . The defense is formulated as a minimax optimization problem:

Objective: Minimize $I(A; x)$ (privacy leakage) while maximizing $I(A; y)$ (utility for next-token prediction), subject to latency constraints.
Architecture (Privacy Adapters): The core mechanism involves inserting lightweight, trainable privacy adapters between frozen transformer layers on the client side. Unlike LoRA-style adapters that add residual connections, these adapters implement a strict information bottleneck:
- They compress activations from dimension $d$ to a low-dimensional subspace $r$ ( $r \ll d$ ) via a down-projection, non-linearity, and up-projection.
- Crucially, they replace the activation rather than perturbing it additively. The authors prove theoretically that injective residual connections (common in other defenses) cannot reduce mutual information, whereas non-injective bottlenecks enforce lossy compression, genuinely reducing MI.
Optimization: The system employs an alternating training schedule. An adversary network (a token classifier) is trained to maximize reconstruction accuracy (minimizing cross-entropy), while the adapters are updated to minimize a combined loss of the adversary's cross-entropy and the downstream task's perplexity. This directly estimates and minimizes a variational upper bound on MI.

Key Contributions

Information-Theoretic Minimax Framework: The paper establishes a formal defense against prompt inversion by explicitly minimizing mutual information via a learned adversary, bridging the gap between theoretical privacy bounds and practical implementation.
Privacy Adapters with Theoretical Guarantees: The authors introduce adapter modules that enforce lossy compression. They provide a formal proof (Theorem 4.2) that residual-style defenses (injective mappings) cannot reduce mutual information, justifying the necessity of their non-injective bottleneck design. They also derive bounds on reconstruction error as a function of the bottleneck dimension.
Sensitive Token Analysis: The framework introduces a methodology to distinguish between the recovery of sensitive tokens (domain-specific terms like medical conditions or legal entities) and common structural tokens (stop words, punctuation). The theory predicts and experiments confirm that the bottleneck disproportionately suppresses high-entropy (sensitive) tokens while preserving low-entropy (common) tokens necessary for syntactic coherence.
Comprehensive Evaluation: The method is validated across three LLMs (LLaMA-2-7B/13B, Mistral-7B) and three domains (Medical, Legal, Airline reviews) against both classification-based and optimization-based prompt inversion attacks.

Experimental Results

Privacy-Utility Tradeoff: The method achieves a significant reduction in attack success rates. Across various settings, it reduces sensitive token recovery by 15–35% compared to baselines, with the best configurations (e.g., $\lambda=0.5$ ) showing up to a 35 percentage point reduction in sensitive token accuracy compared to common tokens.
Comparison with Baselines: The proposed adapters outperform existing defenses including Gaussian/Laplace noise, PCA, and information-leakage reduction methods (NoPeek, Fisher-based masking). Notably, noise-based methods often face a "privacy-utility cliff" where strong privacy destroys utility (perplexity), whereas the adapters maintain usable perplexity (e.g., PPL < 100) while significantly lowering attack accuracy.
Efficiency: The approach introduces minimal latency overhead (<9%) on edge devices (tested on Jetson Orin Nano) and adds a negligible parameter count (<0.3% of the full model).
Robustness: The defense generalizes across attack types; "fresh" attackers trained from scratch on protected activations achieve similar success rates to the training adversary, confirming the defense is not merely hiding information from a specific co-adapted model.

Significance and Claims
The paper claims to provide the first information-theoretic privacy defense for collaborative LLM inference under prompt inversion attacks. Its primary significance lies in shifting the paradigm from heuristic perturbations to principled, lossy compression that offers provable bounds on reconstruction error. The authors argue that their approach demonstrates that learned, task-aware transformations are essential for practical privacy in edge-cloud LLM systems, enabling sensitive applications (healthcare, legal) to utilize LLMs without exposing raw user prompts. The work establishes that privacy protection need not be uniform; by selectively suppressing high-entropy tokens, the system can protect sensitive content while maintaining the structural integrity required for downstream inference.