Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks

1. Problem Statement

The rapid adoption of Large Language Models (LLMs) in enterprise sectors (healthcare, defense, finance) creates a conflict between capability (requiring massive cloud models) and compliance (requiring data privacy).

• The Dilemma: Organizations must choose between using powerful cloud-hosted models (risking data exposure) or running smaller, less capable local models.
• Limitations of Existing Solutions: Traditional privacy-preserving techniques like Homomorphic Encryption (HE), Secure Multi-Party Computation (MPC), and Trusted Execution Environments (TEE) impose prohibitive computational overhead (100–1000x slowdown), making them impractical for real-time LLM inference.
• The Split Inference Bottleneck: While "split inference" (partitioning the model between local and cloud) offers a middle ground, applying it to autoregressive decoding is inefficient. In standard split inference, every generated token requires a full network round-trip (RTT) to the cloud. Over Wide-Area Networks (WANs) with 80–100ms latency, this limits throughput to ~8–11 tokens/second, which is too slow for interactive applications.

2. Methodology

The authors propose a system that combines asymmetric layer splitting with lookahead decoding to overcome WAN latency while preserving privacy.

A. System Architecture & Layer Partitioning

• Local Device (Trusted): Handles the embedding layer (converting tokens to vectors), the first few transformer layers, the last few transformer layers, and the unembedding/LM head (converting vectors back to tokens).
  • Privacy Guarantee: Raw token IDs and the vocabulary mapping never leave the local device.
• Cloud Server (Untrusted): Handles the middle bulk of transformer layers (e.g., layers 2–29 in a 32-layer model).
  • Data Transmitted: Only high-dimensional intermediate activation vectors (float16, ~8KB per token) are sent over the network. These are abstract geometric representations, not tokens.
• Transport Layer: Uses a WebSocket binary protocol (float16) instead of HTTP/JSON to minimize serialization overhead and eliminate per-request TCP handshakes.
• KV-Cache Management: The Key-Value (KV) cache is maintained entirely on the cloud server to avoid transferring large state data across the network.

B. Decoding Strategy: Lookahead Decoding

To address the latency bottleneck, the system adapts Lookahead Decoding (originally designed for GPU parallelism) for network latency amortization:

• Mechanism: Instead of generating one token per RTT, the system uses Jacobi-style parallel iterations to generate a block of candidate tokens. It caches n-gram candidates from these intermediate trajectories.
• Amortization: In a single network round trip, the system verifies multiple speculative tokens. If an n-gram match is found, multiple tokens are committed simultaneously.
• Result: This effectively "hides" the network latency behind the computation of multiple tokens, increasing the effective throughput.

C. Privacy Model

• Threat Model: Semi-honest cloud server (faithfully executes code but tries to infer input from activations).
• Defense: The cloud lacks the embedding matrix and the specific weights of the local layers (which can be fine-tuned/LoRA-adapted). Without these, reconstructing the original input from intermediate activations is computationally difficult.

3. Key Contributions

1. Practical Privacy-Aware System: A deployable split inference system achieving 8.7–9.3 tokens/second on Mistral 7B over an ~80ms WAN link, with projections of 15–19 tok/s at 20ms RTT.
2. Lookahead Decoding Integration: The first application of lookahead decoding to split inference, successfully amortizing network RTT across 1.2–1.5 tokens per step (up to 5x on code).
3. Empirical Privacy Evaluation: Quantified the trade-off between split depth and privacy. An inversion attack recovers ~59% of tokens at a 2-layer split but drops to ~35% at an 8-layer split, with minimal throughput penalty.
4. Scaling Validation: Demonstrated generalization to Mistral NeMo 12B (40 layers) using only 4.9 GB of local VRAM, achieving throughput comparable to the 7B model despite the larger parameter count.
5. Quality Verification: Formally verified that lookahead decoding produces token-identical output to sequential greedy decoding, ensuring zero quality degradation.
6. RTT Decomposition Model: Introduced a model separating wall time into Network RTT and Fixed Overhead, enabling accurate throughput projections across different network conditions (<6.2% error).
7. Deployment Engineering: Highlighted that networking architecture (e.g., direct SSH vs. proxy routing) impacts performance more than geographic distance, providing essential engineering guidelines for real-world deployment.

4. Experimental Results

• Performance:
  • Mistral 7B: 8.7–9.3 tok/s (80ms RTT) with lookahead; up to 13–17 tok/s on lower-latency links.
  • Mistral NeMo 12B: 7.8–8.7 tok/s (80ms RTT).
  • Scaling: The 12B model achieved similar throughput to the 7B model because the network RTT dominates the latency, and the cloud compute time for the extra layers was negligible relative to the RTT.
• Acceptance Rates:
  • Average acceptance rate: 1.21–1.25 tokens/step.
  • Code: Highest acceptance (1.43–1.57 tokens/step) due to repetitive patterns.
  • Creative Text: Lowest acceptance (1.06–1.12 tokens/step).
• Privacy vs. Performance: Increasing local layers from 2 to 8 reduced token recovery accuracy from ~59% to ~35% while adding only ~3ms of local compute latency per token (negligible compared to ~80ms RTT).
• Provider Comparison: Direct SSH access (RunPod) yielded 2–4x lower latency and significantly higher throughput compared to proxy-routed connections (VAST.ai), regardless of geographic proximity.

5. Significance and Implications

• Viability of Split Inference: This work proves that split inference is not just theoretically possible but practically viable for interactive use over public internet connections, provided speculative decoding is used.
• Architectural Privacy: It offers a "good enough" privacy guarantee for organizations that cannot use cryptographic methods (due to performance) but cannot send plaintext to the cloud. It ensures raw text never leaves the device.
• Cost-Effectiveness: By utilizing consumer-grade local GPUs (RTX 3090) and standard cloud GPUs (A100/4090), it lowers the barrier to entry for privacy-preserving LLM deployment.
• Future Direction: The paper establishes a foundation for "compliance-locked" cloud services, where providers can offer high-performance LLMs to regulated industries by partnering with local inference SDKs.

Limitations: The system relies on architectural privacy rather than cryptographic guarantees. If the cloud provider possesses the exact local layer weights (e.g., unmodified public models), inversion attacks remain a risk. Additionally, performance is highly sensitive to network provider architecture (proxy vs. direct).