When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs

1. Problem Statement

Large Language Models (LLMs) increasingly rely on Speculative Decoding to improve inference latency and throughput. This technique uses a smaller "draft" model or heuristics to predict multiple future tokens, which are then verified in parallel by the larger target model. While efficient, this process introduces a new privacy vulnerability:

• The Core Issue: The number of tokens generated per iteration varies depending on whether the speculative predictions were correct or incorrect.
  • Correct Speculation: Multiple tokens are verified and sent in a single iteration (larger packet).
  • Incorrect Speculation: Only one token is generated via standard auto-regressive decoding (smaller packet).
• The Threat: A network-based adversary (e.g., a malicious ISP or compromised router) can observe encrypted network packet sizes or inter-arrival times. Since packet size correlates directly with the number of tokens generated per iteration, the adversary can infer the speculation pattern (the sequence of correct vs. incorrect guesses).
• The Consequence: These speculation patterns are unique to specific input prompts and responses. An attacker can use these patterns to fingerprint user queries, leak private data (e.g., medical symptoms), and even extract confidential data from the model's internal datastores used for speculation.

2. Methodology

The authors propose and evaluate two primary attack vectors and several mitigation strategies.

A. Attack Vectors

1. Query Fingerprinting (Network Adversary):

   • Mechanism: The attacker profiles the "trace" of token counts per iteration for a set of known prompts offline. They train a classifier (Random Forest) to map packet size variations to specific prompts.
   • Execution: In the online phase, the attacker intercepts encrypted traffic, extracts the packet size trace, and uses the classifier to identify the user's private query.
   • Scenarios Tested:
     • Exact Knowledge: Training and testing on the same set of 50 prompts.
     • Constrained: Training on prompts with identical structure but different semantics.
     • Approximate Knowledge: Training on semantically similar but distinct prompts (e.g., rephrased questions).
     • Out-of-Distribution (OOD): Training on public data (e.g., common diseases) to attack specific user queries.

2. Datastore Leakage (Malicious User):

   • Target: Techniques like REST (Retrieval-based Speculative Decoding) that use a datastore of previous interactions to draft tokens.
   • Mechanism: A malicious user crafts inputs to force the model to retrieve specific sequences from the datastore. By observing which tokens are successfully speculated (indicated by larger packet sizes), the attacker can iteratively reconstruct the contents of the confidential datastore.

B. Experimental Setup

• Models & Techniques: Evaluated four speculative decoding schemes: LADE (Lookahead Decoding), REST, BiLD (Big Little Decoder), and EAGLE.
• Datasets: Used medical chatbot datasets (MedAlpaca) and insurance-related queries.
• Environments: Tested on local GPUs (RTX 4090, A100) and remote vLLM servers to simulate real-world deployment.
• Metrics: Attack accuracy (fingerprinting prompts), leakage rate (tokens/sec), and payload overhead for mitigations.

3. Key Results

Query Fingerprinting Accuracy

The attacks achieved significantly high accuracy, far exceeding random guessing (2% baseline for 50 classes):

• Temperature 0.3 (Deterministic):
  • REST: 100% accuracy.
  • BiLD: 95.2% accuracy.
  • LADE: 91.6% accuracy.
  • EAGLE (Remote vLLM): 77.6% accuracy.
• Temperature 1.0 (Stochastic):
  • Accuracy remained high for REST (99.6%) and moderate for others (LADE 61.2%, BiLD 63.6%, EAGLE 24%), still vastly superior to random chance.
• Approximate Knowledge: Even when the attacker only had semantically similar training data, they could identify the topic (e.g., specific disease) with 20–40% accuracy.
• Robustness: The attack outperformed prior timing-based side-channel attacks (which rely on inter-arrival times) because packet sizes are less susceptible to network congestion and server load noise.

Datastore Leakage

• Using the REST technique, a malicious user successfully leaked confidential data from a 120k-conversation datastore.
• Leakage Rate: Up to 25 tokens per second.
• Efficiency: The "Reusing Leaked Sequences" strategy was the most effective, allowing the attacker to extract long sequences by chaining previously leaked tokens.

Mitigation Effectiveness

The authors proposed and evaluated two defenses:

1. Token Aggregation: Waiting to send multiple iterations of tokens in a single packet.
   • Result: Reduced attack accuracy by up to 50% but increased user-perceived latency.
2. Packet Padding:
   • Random Padding: Adding random bytes to packets. Reduced accuracy to near-random levels (2–8%) with a 5–8× increase in payload size.
   • Fixed Padding: Padding all packets to a maximum size (e.g., 1024 bytes).
   • Result: Reduced attack accuracy to ~2% (random guessing) but increased payload size by 230×, which is likely prohibitive for production systems.

4. Key Contributions

1. Discovery of a New Side Channel: Identified that speculative decoding creates input-dependent variations in token counts per iteration, which leak via network packet sizes.
2. High-Accuracy Fingerprinting: Demonstrated that an adversary can recover exact user queries with >90% accuracy across multiple speculative decoding schemes and deployment environments (local and remote).
3. Datastore Extraction: Showed that speculative decoding mechanisms relying on retrieval (like REST) can be exploited to exfiltrate private training data or user interaction logs at high speeds.
4. Comprehensive Defense Analysis: Evaluated trade-offs between security and performance, showing that while perfect security is possible (via massive padding), it comes at a steep cost to bandwidth and latency.

5. Significance

This paper highlights a critical, previously overlooked privacy risk in the deployment of high-performance LLMs. As companies increasingly adopt speculative decoding to reduce latency and costs, they inadvertently expose user inputs to network-level eavesdropping.

• Privacy Implications: In sensitive domains like healthcare (medical chatbots) or finance, this side channel could violate regulations (e.g., HIPAA, GDPR) by leaking specific patient conditions or financial queries.
• Intellectual Property: The ability to leak datastore contents threatens proprietary data used to fine-tune or retrieve-augment models.
• Future Directions: The findings suggest that current "streaming" protocols for LLMs are insecure by default. Future inference frameworks must integrate privacy-preserving mechanisms (like intelligent padding or batched transmission) that do not rely solely on encryption, as encryption does not hide packet size metadata.