💻 computer science

Multi-Tenant Edge-Cloud Hybrid LLM Serving using Speculative Decoding and Quantization

This paper proposes a multi-tenant edge-cloud hybrid LLM serving architecture that combines W4A16-quantized speculative decoding with asynchronous communication and a multi-tenant verification orchestrator to simultaneously address privacy, latency, and resource utilization challenges.

Original authors: Jui-Yu Lin

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jui-Yu Lin

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain (a Large Language Model) that can write stories, solve math, and chat like a human. But there's a catch: this brain is so huge it won't fit in your pocket, and if you try to carry it everywhere, your phone battery dies instantly.

So, we have two bad choices:

The Cloud: Send your questions to a giant supercomputer far away. It's smart, but the message takes a long time to travel there and back (like sending a letter by carrier pigeon), making the chat feel slow and clunky. Plus, you have to trust the pigeon with your private secrets.
The Edge: Try to run the whole brain on your phone. It's fast and private, but your phone isn't strong enough, and the answers might be a bit silly because the model had to be shrunk down too much.

This paper suggests a clever middle ground: a team-up between your phone and the cloud that works like a high-speed relay race with a twist.

The Team-Up: The "Drafting" Sidekick and the "Verifying" Boss

Here's how the new system works, according to the authors:

1. The Edge Sidekick (Your Phone)
Instead of waiting for the cloud to answer, your phone runs a tiny, super-compact version of the robot brain. The authors suggest using a specific "shrink-wrap" technique called W4A16 quantization. Think of this as compressing a high-definition movie into a tiny file that still looks good enough to understand the plot.

The Magic: This tiny version is smart enough to guess the next few words (tokens) in a sentence very quickly. The paper notes that while shrinking the model does make it slightly less accurate (the "perplexity" score goes from 5.47 to about 5.74 or 5.83), it's still good enough to make a solid guess.
The Rule: The authors explicitly argue against shrinking it even more (like W4A4). They say that if you compress it too much, the guesses get so bad that the system wastes time correcting them, slowing everything down. So, they stick to the "just right" W4A16 size.

2. The Cloud Boss (The Supercomputer)
While your phone is busy guessing the next few words, the cloud is doing the heavy lifting. It holds the full-size, perfect robot brain. Its job isn't to start from scratch; it just has to verify what the phone guessed.

The Twist: In old systems, the phone would guess, then stop and wait for the cloud to say "Yes" or "No." This is called "mutual waiting," and it's like a game of tennis where you wait for the ball to come back before you even swing your racket.
The New Move: This paper proposes an asynchronous (non-blocking) protocol. The phone keeps guessing the next words while the cloud is still checking the previous ones. It's like a conveyor belt where the phone keeps packing boxes while the cloud is stamping the ones already on the belt. This hides the travel time (network latency) so you don't feel the delay.

3. The Multi-Tenant Orchestra
Here's the second big trick. Usually, if 100 people are using the cloud, the computer gives each person their own tiny, empty room. This is wasteful.
The authors suggest a Multi-Tenant Orchestrator. Imagine a busy restaurant kitchen. Instead of giving every customer their own private chef (who sits around waiting for an order), the kitchen has one super-efficient team that takes orders from everyone at once.

The cloud groups requests from many different phones into one big "batch" to check all at once.
This keeps the cloud's powerful graphics cards (GPUs) working at 100% capacity, rather than sitting idle while waiting for one person to type.

What the Math Says (But Keep It Simple)

The authors built a mathematical model to see if this idea holds up. They didn't run a massive real-world test with thousands of people yet; instead, they used formulas to simulate how the system should behave.

The Speed Limit: They calculated that if the cloud is fast enough to check a batch of guesses while the phone is making the next batch, the internet delay disappears from the equation.
The Bottleneck: The system works best when the cloud isn't overwhelmed. If too many people join the party, the cloud gets stuck in a "queuing delay." The model suggests that by carefully balancing how many people are on the system, you can keep the speed high.
The Result: In their simulations, this setup suggests that the system can achieve speeds close to running the model entirely on your phone, but with the accuracy of the giant cloud brain, all while keeping your private data mostly on your device.

What This Paper Is Not Saying

It's important to know what this paper doesn't claim:

It hasn't been proven in a real-world deployment with thousands of users yet. The authors are suggesting this architecture based on theoretical models and existing tools (like llama.cpp for phones and vLLM for the cloud).
It doesn't claim to solve every privacy issue. While it keeps raw prompts local, it still sends some speculative data to the cloud.
It doesn't say this works for any model size. The math relies on specific conditions where the cloud is fast enough to keep up with the phone's guessing speed.

The Bottom Line

The authors propose a way to make AI chat feel instant and private without needing a supercomputer in your pocket. By letting your phone make quick, "good enough" guesses and having the cloud check them in a busy, efficient group, they suggest we can bypass the slow internet delays that usually ruin the experience. It's a promising blueprint for the future, suggesting that if we build the right "relay race" team, we can have the best of both worlds.

Technical Summary: Multi-Tenant Edge-Cloud Hybrid LLM Serving using Speculative Decoding and Quantization

1. Problem Statement

The deployment of Large Language Models (LLMs) faces a fundamental trade-off between edge inference and cloud inference.

Edge Inference: Preserves user privacy and reduces infrastructure costs but is severely constrained by the memory and compute capabilities of edge hardware.
Cloud Inference: Offers high computational power but introduces significant network latency, risks of data leakage during transmission, and inefficient resource utilization due to the intermittent nature of user requests.

Existing hybrid architectures attempt to bridge this gap but often suffer from a "mutual waiting" bottleneck. In traditional distributed speculative decoding, the edge client must block execution while waiting for the cloud server to verify generated tokens. If the Wide-Area Network (WAN) Round-Trip Time (RTT) exceeds the cloud's execution time, the system idles, negating latency benefits. Furthermore, dedicating cloud instances to individual edge users leads to systemic resource fragmentation and low GPU utilization.

2. Methodology

The paper proposes a Multi-Tenant Hybrid Inference Framework that integrates three core techniques: Quantized Speculative Decoding, Asynchronous Communication, and Multi-Tenant Verification Orchestration.

2.1 Edge-Side Quantized Drafting

To address edge memory constraints, the system employs a W4A16 quantized draft model (4-bit weights, 16-bit activations) running locally on the edge device.

Rationale: While extreme quantization (e.g., W4A4) reduces memory further, it causes significant perplexity (PPL) degradation, lowering the token acceptance rate.
Selection: The authors analyze LLaMA-2 7B on WikiText-2, showing that W4A16 (via OmniQuant or GPTQ) incurs only a 5–7% accuracy drop (PPL ~5.74–5.83 vs. 5.47 baseline) compared to the baseline. This configuration maintains sufficient alignment with the cloud target model to ensure a high acceptance rate while drastically reducing the memory footprint.

2.2 Asynchronous Communication Protocol

To eliminate the "mutual waiting" bottleneck, the framework replaces synchronous blocking with a non-blocking, asynchronous protocol inspired by PicoSpec.

Mechanism: The edge device continuously drafts token sequences based on local top-1 predictions without waiting for cloud verification.
Pipeline Decoupling: The edge generates the next block of tokens ( $k+1$ ) while the cloud verifies the previous block ( $k$ ). This overlaps edge drafting time ( $T_{draft}$ ) and network transit ( $T_{network}$ ) with cloud verification time ( $T_{verify}$ ), effectively masking WAN latency.

2.3 Multi-Tenant Cloud Verification Orchestrator

To address cloud-side resource idleness, the system introduces a multi-tenant orchestrator that aggregates verification requests from heterogeneous edge devices.

Architecture: Instead of dedicated sessions, the cloud treats the target model as a globally shared resource.
Scheduling: Using principles from decentralized data flows (e.g., Pathways) and LLM-native serving (e.g., Orca, chunked prefill), the orchestrator multiplexes concurrent requests into single hardware batches.
Benefit: This ensures GPU saturation even if individual tenants experience transient network delays, preventing localized stalls from propagating latency penalties to the entire system.

3. Theoretical Modeling

The paper formulates the performance trade-offs mathematically to demonstrate the system's efficiency.

Speculative Acceptance: The expected number of accepted tokens $E[A]$ is derived from the acceptance probability $\alpha$ . The authors argue that W4A16 is critical to keeping $\alpha$ high, as aggressive quantization would exponentially reduce $E[A]$ .
Latency Analysis:
- Synchronous Latency: $T_{sync} = \gamma \cdot T_{draft} + T_{network} + T_{verify}$ (strictly additive).
- Asynchronous Latency: $T_{async}$ is bounded by the maximum of the parallel stages. The system achieves optimal performance when $T_{verify} \approx \gamma \cdot T_{draft} + T_{network}$ , creating a balanced pipeline where network overhead is hidden.
Throughput Bounds: Cloud verification latency is modeled using the Roofline model, bounded by either memory bandwidth or computational throughput. The orchestrator aims to keep the batch size $B$ optimal to saturate the GPU without incurring excessive queuing delays ( $T_{queue}$ ).

4. Key Contributions

The paper claims three primary contributions:

WAN Latency Masking: The introduction of a non-blocking communication protocol that decouples edge drafting from cloud verification, overcoming the mutual-waiting problem inherent in distributed speculative decoding.
Throughput Maximization: A multi-tenant cloud orchestrator that aggregates verification requests from multiple edge devices, maximizing GPU utilization and preventing resource fragmentation.
Privacy-Preserving Inference: A design that minimizes privacy exposure by offloading only partial speculative data (draft tokens) rather than raw user prompts to the cloud.

5. Results and Evaluation Strategy

The paper is primarily an architectural proposal and theoretical analysis rather than a report on empirical experimental results.

Theoretical Bounds: The analysis demonstrates that under optimal conditions (where the pipeline is locally bounded by edge performance), the system can eliminate network transit delays and approach the edge drafting speed as the Inter-Token Latency (ITL).
Proposed Evaluation: The authors outline a future evaluation strategy to benchmark the system against cloud-only pipelines and synchronous distributed speculation. Key metrics include:
- Inter-Token Latency (ITL): To validate latency masking.
- Token Acceptance Rate ( $\alpha$ ): To confirm the alignment between W4A16 drafts and FP16 targets.
- Cloud Throughput: To measure aggregate generation rates under high concurrency.
Implementation Blueprint: The paper proposes a practical stack using llama.cpp/MLC-LLM for edge drafting, vLLM (with PagedAttention) for cloud verification, and Ray for asynchronous orchestration.

6. Significance and Claims

The paper positions this framework as a scalable and privacy-preserving path for edge-cloud collaborative LLM deployment.

Decoupling: It effectively decouples edge execution from cloud handshake loops, eliminating the mutual waiting bottleneck that has hindered previous hybrid approaches.
Scalability: The theoretical analysis indicates that the system scales efficiently under multi-tenant workloads by treating the cloud target model as a shared resource graph rather than a set of blocking sessions.
Balance: It offers a solution that harmonizes deployment efficiency, privacy, and inference throughput, breaking the traditional trade-off where improving one metric degrades the others.

The authors conclude that while the framework theoretically eliminates network delays and achieves edge-generation speeds, future work is required to validate these performance results through end-to-end practical deployment using distributed serving infrastructures.