💬 NLP

What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference

This paper exposes critical privacy vulnerabilities in split inference for large language models by introducing ActInv, a method that reconstructs client inputs from intermediate activations despite common defenses, and proposes the Perturbation Amplification Factor (PAF) metric and the PriPert defense to systematically analyze and mitigate these leakage risks.

Original authors: Mingyuan Fan, Yu Liu, Fuyi Wang, Cen Chen

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingyuan Fan, Yu Liu, Fuyi Wang, Cen Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Split" Dilemma

Imagine you have a very smart, giant robot brain (a Large Language Model or LLM) that lives in a powerful cloud server. You have a small, weak smartphone that can't run this giant brain on its own.

To solve this, you use Split Inference. Think of it like a relay race:

You (the Client): You run the first few meters of the race. You take your secret question (e.g., "What's wrong with my knee?") and process it just enough to get a "baton" (intermediate data) ready.
The Server: You hand the baton to the giant robot. It runs the rest of the race, finishes the answer, and sends it back to you.

The idea is that by only sending the "baton" instead of your raw question, you keep your secrets safe. The paper asks: Is the baton actually safe, or can the server read your mind just by looking at it?

The Attack: "ActInv" (The Mind Reader)

The researchers discovered that the "baton" is not safe. They created a tool called ActInv that acts like a high-tech detective.

The Analogy: Imagine you send a server a blurry, scrambled photo of your face (the intermediate data). A normal person might think, "I can't tell who that is." But ActInv is like a super-sophisticated AI that takes a blank canvas and starts painting a face. It keeps adjusting the paint until the blurry photo it creates matches the scrambled photo you sent perfectly. Once it matches, it knows exactly what your original face (your secret question) looked like.
The Result: The paper shows that ActInv is terrifyingly good. It can reconstruct your original question with over 98% accuracy, even if you try to hide it by adding static noise or blurring parts of the data. It's like trying to hide a message in a snowstorm, but the detective has a thermal camera that sees right through the snow.

Why Does This Happen? (The "Weak Links")

The researchers wanted to know why the server could read the mind so easily. They invented a metric called PAF (Perturbation Amplification Factor).

The Analogy: Think of the AI model as a long tunnel with many rooms (layers). Some rooms are made of thick, soundproof concrete (high PAF). If you whisper in there, the sound dies out, and the person at the end can't hear you. Other rooms are made of thin glass or amplifiers (low PAF). If you whisper in there, the sound gets louder and clearer by the time it reaches the end.
The Discovery: They found that the "glass rooms" (specifically the activation layers where the AI makes decisions) are surprisingly weak. They don't scramble the information; they actually help the detective reconstruct the original input. Even though these layers are supposed to make the AI "smart," they accidentally make it "leaky."

The Failed Defenses

Before this paper, people thought adding "noise" (like static on a radio) or "sparsifying" (hiding some data points) would stop the detective.

The Reality: The paper shows these defenses are like trying to stop a hurricane with an umbrella. The detective (ActInv) is so smart it can filter out the noise and still see the original message. The only way to stop it with these methods is to add so much noise that the AI stops working entirely, which ruins the service for everyone.

The Solution: "PriPert" (The Smart Shield)

Since simple noise doesn't work, the researchers designed a new defense called PriPert.

The Analogy: Instead of throwing random sand in the detective's eyes (random noise), PriPert is like a martial artist who knows exactly where to strike. It calculates the most sensitive direction in the data.
- Imagine the data is a balloon. If you poke it randomly, it might just wobble. But if you poke it in the exact spot where it's most fragile, it pops.
- PriPert finds that "fragile spot" in the data and adds a tiny, targeted push. This pushes the detective's reconstruction off-course, making them guess the wrong question, while keeping the balloon (the AI's answer) intact enough to still be useful.
The Result: PriPert is much better than the old methods. It successfully confuses the detective, making it impossible to guess the original question, while still letting the AI give you a helpful answer.

Summary of Findings

The Risk: Split inference (sending partial data to a server) is currently very unsafe. A curious server can easily reconstruct your private questions.
The Cause: Certain parts of the AI model act like amplifiers, making it easy to reverse-engineer the input.
The Fix: Simple noise doesn't work. You need smart, targeted noise (PriPert) that specifically targets the weak points of the model to break the reconstruction without breaking the AI's ability to answer questions.

In short: If you are using split inference today, your secrets are likely visible to the server. But there is a new, smarter way to hide them that doesn't break the system.

Technical Summary: What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference

1. Problem Statement

The deployment of Large Language Models (LLMs) on resource-constrained devices is hindered by computational and memory demands. Split inference has emerged as a solution, partitioning the model between a client (holding initial layers) and a server (holding the remainder). The client transmits only intermediate activations rather than raw inputs, theoretically preserving privacy. However, the authors argue that the assumption that intermediate activations inherently preserve privacy is unproven. This paper investigates whether an "honest-but-curious" server, possessing full white-box access to the model architecture, can reconstruct the client's sensitive input prompts from these intermediate activations.

2. Methodology

2.1 Attack Design: ActInv

To quantify privacy leakage, the authors propose ActInv, an inversion attack that reconstructs the client's raw input sequence from the intercepted intermediate activations ( $h_{Q1}$ ).

Mechanism: Unlike sequential token recovery methods, ActInv formulates reconstruction as a continuous optimization problem over the input embedding space. It initializes a random continuous embedding vector $\hat{h}_0$ and uses gradient-based optimization (e.g., Adam) to minimize the distance between the forward pass of the client-side submodel ( $F_C(\hat{h}_0)$ ) and the observed activations ( $h_{Q1}$ ).
Discretization: Once the continuous embeddings are optimized, they are projected back to discrete tokens via nearest-neighbor search within the model's vocabulary embedding matrix.
Efficiency: By optimizing the full sequence simultaneously rather than token-by-token, ActInv achieves a time complexity of $O(Tn^2)$ , offering a significant computational advantage over autoregressive methods like SipIt ( $O(Tn^3)$ ).

2.2 Vulnerability Analysis: Perturbation Amplification Factor (PAF)

To understand why certain layers are vulnerable, the authors introduce PAF, a metric quantifying a layer's inherent resistance to reconstruction.

Definition: PAF measures how much a small perturbation in a layer's output ( $\delta$ ) translates to uncertainty in the inferred input ( $\Delta$ ). It is derived from the Jacobian matrix of the layer.
Insight: A high PAF indicates that small output perturbations cause large input deviations (making reconstruction difficult), while a low PAF indicates the layer is "stable" and easily inverted.
Findings: The analysis reveals non-uniform vulnerability across layers. Specifically, non-linear activation layers (e.g., Sigmoid/SwiGLU) and certain projection layers often exhibit low PAF, making them "weak links" where naive noise defenses fail to induce significant reconstruction errors.

2.3 Defense Design: PriPert

Based on the PAF analysis, the authors propose PriPert, a defense mechanism that injects adversarially calibrated perturbations into intermediate activations.

Strategy: Instead of adding isotropic (random) noise, PriPert solves a constrained optimization problem to inject perturbations along the directions that maximize the reconstruction error (aligned with the layer's most sensitive directions) while minimizing the degradation of the server-side output (utility).
Implementation: The defense utilizes a path integral approximation to estimate the input deviation caused by the perturbation. The authors find that an $L_0$ -constrained solution (sparse perturbations) often outperforms $L_2$ -constrained solutions, as it avoids preserving sign information that adversaries might exploit.

3. Key Results

3.1 Attack Effectiveness (ActInv)

High-Fidelity Reconstruction: ActInv achieves Precision and Recall exceeding 98% across various models (Qwen3, Falcon3) and datasets (AlpacaEval, iCliniq) without any defense.
Resilience to Naive Defenses: Common defenses like Gaussian noise injection and activation sparsification are largely ineffective. Even with high sparsity ratios (e.g., 0.7) or moderate noise, ActInv maintains high reconstruction accuracy. Only extreme noise levels (which destroy model utility) significantly hinder the attack.
Efficiency: ActInv is computationally efficient, reconstructing a 100-token query in approximately 20 seconds on a single GPU, significantly faster than state-of-the-art sequential attacks.

3.2 Vulnerability Insights (PAF)

Layer Variance: PAF values vary significantly across layers and models. For instance, activation layers consistently show low PAF, confirming they are primary sources of leakage.
Directional Sensitivity: The "Max-PAF" (sensitivity to aligned perturbations) is often significantly higher than the "Expected PAF" (sensitivity to random noise), validating that random noise is a suboptimal defense strategy.

3.3 Defense Performance (PriPert)

Privacy Improvement: PriPert significantly reduces reconstruction quality. At a sparsity ratio of 0.7, ActInv's Precision drops below 30% (compared to >60% for standard sparsification), and ROUGE-L scores plummet, indicating the semantic content is effectively obscured.
Utility Preservation: Unlike naive defenses that collapse model utility at high perturbation levels, PriPert maintains acceptable utility at moderate sparsity levels (e.g., 0.5). The authors note that "Accuracy & Correctness" are more sensitive to perturbations than "Clarity & Style," but overall scores remain usable.
Robustness: PriPert remains effective against other state-of-the-art inversion attacks (A1, TBS, PIA), reducing their reconstruction scores by large margins.

4. Significance and Contributions

The paper claims the following contributions:

Exposing the Vulnerability: It provides the first comprehensive investigation demonstrating that split inference for LLMs does not inherently guarantee privacy. The authors show that an honest-but-curious server can reconstruct user inputs with high fidelity using a relatively simple, efficient optimization-based attack (ActInv).
Theoretical Metric (PAF): It introduces PAF to systematically quantify and explain the varying susceptibility of different model layers to inversion, revealing that non-linear activation layers are surprisingly vulnerable.
Effective Defense (PriPert): It proposes and validates PriPert, a defense that leverages the directional sensitivity of layers to maximize privacy leakage prevention while minimizing utility loss. The results suggest that adversarially calibrated perturbations are superior to naive random noise or sparsification.
Practical Implications: The work highlights a critical tension in split inference: standard defenses are insufficient, and effective privacy protection requires careful calibration of perturbations to avoid destroying the utility of the LLM.

The authors conclude that while split inference offers computational benefits, it requires robust, non-naive privacy mechanisms to be viable for sensitive domains like healthcare. Future work is suggested in designing privacy-aware compression and privacy-sensitive activation functions.