⚡ electrical engineering

StarSD: One-for-Many Speculative Decoding

StarSD is a scalable, one-for-many speculative decoding framework that utilizes a star topology to decouple drafting and verification across distributed nodes, enabling a single draft model to efficiently serve multiple target models while improving resource utilization and latency in heterogeneous LLM inference clusters.

Original authors: Junhao He, Feiran You, Hongyang Du

Published 2026-01-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junhao He, Feiran You, Hongyang Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a high-end restaurant (the Target Model) that serves complex, gourmet dishes. To speed up service, you hire a sous-chef (the Draft Model) who is faster but less experienced. The sous-chef guesses what the next few ingredients should be and writes them down on a notepad. The head chef then quickly checks these guesses. If the guesses are right, the head chef accepts them and moves on; if they are wrong, the head chef corrects them.

In the past, this team worked in a very specific way: the sous-chef and the head chef were stuck in the same tiny kitchen. They had to share the same limited counter space (memory). If the head chef was busy checking the list, the sous-chef had to stand around doing nothing, waiting for the head chef to finish. This created a lot of "dead time" where the kitchen wasn't fully productive.

StarSD is a new way of organizing this kitchen that solves two big problems: running out of counter space and wasting time while waiting.

The Problem: The "One-to-One" Bottleneck

Traditionally, every head chef had their own dedicated sous-chef.

Space Issue: In modern restaurants, the head chefs are huge and need a lot of counter space. There often isn't enough room left for a dedicated sous-chef to stand next to them.
Idle Time: When the head chef is checking the list, the sous-chef sits idle. When the sous-chef is writing, the head chef sits idle. They take turns, leaving the kitchen half-empty half the time.

The StarSD Solution: The "One-for-Many" Star

StarSD changes the layout. Instead of every head chef having their own sous-chef, you have one super-efficient sous-chef who serves many head chefs at once.

Here is how it works, using the paper's logic:

1. The Star Topology (The Hub and Spoke)
Imagine the sous-chef is a central hub in the middle of the kitchen. The head chefs are scattered around the room (on different counters or even in different rooms).

The head chefs shout out their current order (the "verified prefix").
The central sous-chef listens to all of them. As soon as one head chef is ready, the sous-chef immediately starts guessing the next few ingredients for that chef.
While the sous-chef is guessing for Chef A, Chef B might be checking Chef A's list. Meanwhile, Chef C is ready for a new guess. The sous-chef never stops working because there is always someone waiting for a guess.

2. Killing the "Idle Gaps"
In the old system, if the head chef took 10 seconds to check a list, the sous-chef sat idle for 10 seconds.
In StarSD, the sous-chef doesn't wait. While Chef A is checking, the sous-chef immediately turns to Chef B, then Chef C. By the time Chef A is done, the sous-chef has already written guesses for B and C. The sous-chef is now "work-conserving," meaning they are constantly busy, which makes the whole kitchen run much faster.

3. The "One-for-Many" Magic
The paper calls this "One-for-Many." One draft model (the sous-chef) serves many target models (the head chefs).

Memory Savings: You don't need to squeeze a sous-chef into every single head chef's kitchen. You just need one central sous-chef station. This frees up space for more head chefs to work.
Speed: Because the sous-chef is constantly working without stopping to wait, the "guessing" part of the process becomes smoother and faster.

The Two Stages of Performance

The paper explains that this system behaves in two ways depending on how busy the restaurant is:

Stage 1: The "Under-loaded" Phase (Few Chefs)
If you only have 2 or 3 head chefs, the central sous-chef might still have to wait a little bit because the chefs aren't shouting orders fast enough. The kitchen is running, but not at full speed. Adding more chefs helps fill the gaps.
Stage 2: The "Fully-loaded" Phase (Many Chefs)
Once you have enough chefs (the paper suggests around 4 or more in their tests), the sous-chef is so busy that they never stop. The kitchen runs at maximum efficiency. Adding even more chefs doesn't make the sous-chef faster (they are already maxed out), but it does increase the total number of dishes served by the whole team.

The Catch: The "Travel Time"

Since the head chefs and the sous-chef might be in different rooms (different computers or servers), there is a tiny delay in shouting the order and getting the answer back (communication time).

The paper found that this delay is small enough that the "One-for-Many" system is still much faster than the old "One-to-One" system, even with the travel time.
However, if you add too many chefs, they start waiting in line to talk to the sous-chef. The paper suggests a "sweet spot" where you have enough chefs to keep the sous-chef busy, but not so many that they get stuck in a traffic jam.

Summary

StarSD is like hiring one super-fast, central assistant to help a whole team of experts.

Old Way: Every expert has their own assistant, but they run out of desk space, and the assistant sits idle half the time.
StarSD Way: One assistant runs around helping everyone. They never sit idle because there's always someone else to help. This saves space, keeps the assistant working hard, and gets more work done overall, even if the assistant has to walk between different desks.

The paper proves this works on real computer hardware (GPUs), showing that you can serve more requests faster without changing the "recipes" (the AI models) themselves, just by changing how the kitchen is organized.

Technical Summary: StarSD: One-for-Many Speculative Decoding

1. Problem Statement

Large Language Model (LLM) inference systems face significant pressure on latency, throughput, and hardware efficiency, particularly when serving concurrent requests in shared accelerator clusters. The primary bottleneck is autoregressive decoding, where each token requires a serial forward pass. Speculative decoding has emerged as a lossless acceleration technique that decouples token proposal (via a lightweight draft model, $M_q$ ) from verification (via a large target model, $M_p$ ).

However, existing speculative decoding frameworks predominantly assume a centralized execution mode, where $M_q$ and $M_p$ are co-located on a single accelerator (e.g., a GPU). This design faces critical limitations in modern production environments:

Memory Constraints: As LLMs scale, the combined memory footprint of the target model, the draft model, and the Key-Value (KV) cache often exceeds the capacity of a single accelerator (e.g., an RTX 4090 with 24GB), restricting support for long-context serving.
Edge Deployment: Resource-constrained edge devices (mobile phones, Jetson) cannot practically host both models simultaneously.
Inefficiency in Disaggregation: Simply separating $M_q$ and $M_p$ across different accelerators introduces a new inefficiency. In a standard "draft-then-verify" workflow, when the target model verifies tokens, the draft accelerator remains idle. This alternating execution creates idle gaps, leading to bursty computation that prevents the draft model from maintaining a steady, high-throughput state, thereby increasing per-token latency and variance.

2. Methodology: StarSD Framework

The authors propose StarSD, a one-for-many speculative decoding framework designed for distributed environments. StarSD utilizes a single draft model instance to concurrently serve multiple target model instances across distributed nodes via a star topology.

Core Design Principles

Work-Conserving Execution: StarSD decouples drafting and verification. The draft instance ( $M_q$ ) maintains a global request buffer ( $Q_{in}$ ) and an outgoing reply buffer ( $Q_{out}$ ). It operates in a non-blocking loop, continuously serving pending requests from any of the $N$ active target instances ( $M_p^{(i)}$ ). This ensures that $M_q$ remains busy as long as there is work, effectively amortizing verification and communication latencies across multiple targets.
Per-Target State Isolation: To maintain correctness under concurrency, StarSD isolates per-session states (including KV caches and metadata) using unique tags and dedicated communication ports established during a handshake. This prevents cross-session interference and ensures that the acceptance probability calculations remain valid for each specific prefix.
Asynchronous Multiplexing: The framework multiplexes asynchronous returns from multiple targets. By overlapping the verification and communication phases of one target with the drafting phase of others, StarSD eliminates the idle gaps inherent in single-pair distributed decoding.

System Model and Analysis

The authors develop an analytical system model to characterize performance under different load conditions, separating progress (expected accepted tokens) from time (iteration duration).

Iteration Time ( $T_\gamma$ ): Defined as $T_\gamma = N \cdot S(d) + T_{idle}$ , where $S(d)$ is the service time for $N$ requests and $T_{idle}$ is the waiting time caused by verification delays.
Load Regimes:
- Under-loaded: When the number of targets $N$ is small, $T_{idle} > 0$ . The draft server experiences idle gaps while waiting for verification results.
- Fully-loaded: When $N$ is sufficiently large such that the service time for $N-1$ other targets covers the return time of the current target, $T_{idle}$ vanishes. The draft server becomes work-conserving, maximizing utilization.
Throughput Trade-off: The model identifies a saturation point ( $N_{full}$ ) where adding more targets no longer increases the draft server's service capacity but introduces queueing delays. However, even beyond this point, the system throughput remains higher than standalone inference as long as the per-target throughput does not fall below the standalone baseline.

3. Key Contributions

Framework Proposal: StarSD is the first framework to enable one-for-many distributed speculative decoding, isolating per-target states while enabling work-conserving draft execution in the presence of asynchronous verification returns.
Analytical Model: The authors provide a system model that decouples iteration progress from iteration time, characterizing load regimes, scaling behavior, and the trade-off between per-target throughput and system-wide throughput.
Empirical Validation: Extensive experiments across heterogeneous deployments (intra-machine multi-GPU, inter-machine multi-GPU, and edge-server setups like Computer-Jetson) demonstrate that StarSD simplifies deployment and supports flexible resource allocation without modifying model weights or output distributions.

4. Experimental Results

The paper evaluates StarSD using EAGLE as the draft backend and Vicuna-7B as the target, across various configurations:

Throughput Scaling: In the under-loaded regime (small $N$ ), system throughput increases significantly as $N$ grows because idle gaps are amortized. In the fully-loaded regime (large $N$ ), system throughput saturates near its maximum, while per-target throughput stabilizes.
Latency and Stability: StarSD significantly reduces the per-token compute time and latency variance compared to bursty execution. Experiments show that eliminating idle gaps keeps the draft model in a favorable, steady-state operating regime.
Distributed Overhead: While distributed deployment introduces communication overhead (serialization, network latency), the gains from continuous draft execution and higher memory packing efficiency (allowing more target instances per accelerator) outweigh these costs.
Hardware Agnosticism: The framework performs effectively across heterogeneous hardware (RTX 4090, A100, Jetson), proving its portability.

5. Significance and Claims

The authors claim that StarSD represents a practical and scalable approach to bringing speculative decoding to modern cloud and edge inference infrastructures.

Resource Efficiency: By decoupling drafting from verification, StarSD allows the draft model to be shared as a service, enabling finer-grained memory packing. This allows accelerators to host more target instances or longer context lengths within the same memory budget, addressing the "memory wall" in LLM serving.
Scalability: The framework enables linear scaling of system throughput with the number of target instances until saturation, making it suitable for high-concurrency serving scenarios.
Practicality: StarSD achieves these gains without altering model weights or generation semantics, making it a drop-in improvement for existing speculative decoding pipelines in distributed settings.

The paper concludes that while StarSD introduces a centralized draft service that could become a trust point (requiring strict isolation and security measures), its ability to eliminate idle gaps and improve hardware utilization makes it a viable solution for the next generation of LLM serving systems.