💬 NLP

HiSpec: Hierarchical Speculative Decoding for LLMs

HiSpec is a high-throughput speculative decoding framework that leverages early-exit models for low-overhead intermediate verification and reuses computational states across draft, verifier, and target models to significantly accelerate LLM inference without compromising accuracy.

Original authors: Avinash Kumar, Sujay Sanghavi, Poulami Das

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Avinash Kumar, Sujay Sanghavi, Poulami Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the editor of a massive, high-stakes newspaper. You have a brilliant, slow-thinking senior editor (the Target Model) who produces perfect articles but takes a long time to write and fact-check every single word. You also have a fast, eager junior intern (the Draft Model) who can spit out sentences in a flash, but they often make mistakes or hallucinate facts.

The Old Way: The "Stop-and-Check" Bottleneck

In the traditional method (called Speculative Decoding), the process works like this:

The intern writes a whole paragraph (speculates 5 words).
The senior editor stops everything, reads the whole paragraph, and checks every single word against their own knowledge.
If the intern made a mistake, the editor throws out the whole paragraph and starts over. If they were right, the editor accepts it.

The Problem: The senior editor is so slow at checking that the intern spends most of their time just waiting around. The "checking" part is the bottleneck. In fact, the paper notes that for big models, checking can take 2 to nearly 7 times longer than the intern actually writing the words.

The New Idea: HiSpec (Hierarchical Speculative Decoding)

The authors of this paper, HiSpec, realized that waiting for the senior editor to check everything is wasteful. They proposed a smarter workflow using a three-tier system inside a single model:

The Intern (Draft Layer): A very shallow part of the model that writes words super fast.
The Team Lead (Intermediate Verifier): A "middle-management" layer. This isn't the full senior editor, but it's smart enough to spot obvious mistakes quickly.
The Senior Editor (Target Layer): The full, deep model that gives the final, perfect approval.

How HiSpec Works (The Analogy)

Instead of the intern writing a whole paragraph and then waiting for the senior editor to check it all, HiSpec changes the game:

Step 1: The intern writes a few words.
Step 2: The Team Lead (Intermediate Verifier) immediately glances at them.
- If the Team Lead sees a glaring error: They reject it instantly. The intern doesn't have to wait for the Senior Editor to waste time on it. The intern immediately starts writing the next batch of words.
- If the Team Lead thinks it looks okay: They give a "tentative thumbs up."
Step 3: The Senior Editor only steps in to do a full, deep check on the words the Team Lead approved.
Step 4 (The Safety Net): To make sure the Team Lead didn't miss anything subtle, the Senior Editor does a full check every few words to ensure the final output is 100% perfect.

The Secret Sauce: "Reusing the Notes"

The paper highlights a clever trick to save even more time. Usually, when you check a sentence, you have to re-read the whole context from scratch. HiSpec is like a smart assistant who reuses their notes.

When the intern writes a word, they leave a "sticky note" (Key-Value cache) on the desk. When the Team Lead checks it, they pick up that same note instead of writing a new one. When the Senior Editor checks it, they use the same note again. This means the computer doesn't have to do the heavy lifting of re-calculating the context for every single step.

The Results

The paper claims this approach is a huge win:

Speed: It makes the whole process 1.28 times faster on average, and up to 2 times faster in some cases, compared to the old methods.
Accuracy: Unlike some other "fast" methods that might let errors slip through, HiSpec guarantees the final output is exactly the same as if the slow Senior Editor had written it alone.
Efficiency: It doesn't need to train a whole new "Team Lead" model from scratch; it just uses a specific layer inside the existing model that is already good at this job.

Summary

Think of HiSpec as a fast-track security line at an airport.

Old way: Everyone waits for the head security officer to check every single bag, no matter how small.
HiSpec: A quick scanner (Team Lead) checks for obvious threats first. If it's clear, you move fast. If it looks suspicious, it gets flagged. The head officer (Senior Editor) only does the deep, slow scan on the bags that passed the quick check, and they do it periodically to ensure safety.

The result? You get through the airport (generate text) much faster, but you don't compromise on security (accuracy).

Technical Summary: HiSpec: Hierarchical Speculative Decoding for LLMs

1. Problem Statement

Large Language Model (LLM) inference faces a fundamental trade-off between throughput and accuracy. While speculative decoding improves throughput by using a smaller "draft" model to propose tokens that a larger "target" model verifies, the technique is currently bottlenecked by the verification phase.

The authors identify a "verification wall" where token verification is significantly slower (2× to 6.9×, and up to 10.3× in some configurations) than draft token generation. This latency dominates the total inference time because subsequent tokens cannot be generated until previous tokens are verified. Existing works primarily focus on accelerating the drafting phase, yielding only marginal throughput gains because the verification bottleneck remains unaddressed.

Previous attempts to solve this via "intermediate verification" (e.g., using an auxiliary model to discard inaccurate tokens early, such as in SPRINTER) suffer from three main limitations:

Training Overheads: They require training auxiliary models to approximate the target.
Memory Footprint: Orchestrating multiple models simultaneously increases memory usage.
Accuracy Degradation: Some methods rely on approximate heuristics or skip full target verification, leading to lossy outputs that deviate from the target model's auto-regressive distribution.

2. Methodology: HiSpec

The authors propose Hierarchical Speculative Decoding (HiSpec), a framework designed to accelerate token verification with low overhead while maintaining lossless accuracy. HiSpec leverages Early-Exit (EE) models, which are LLMs explicitly trained to allow tokens to exit at selected intermediate layers, producing meaningful token distributions without traversing the full network.

Core Mechanism

HiSpec utilizes a single EE-model with three distinct operational layers:

Draft Layer ( $L_d$ ): A shallow exit layer used to generate draft tokens.
Intermediate Verifier Layer ( $L_i$ ): A deeper exit layer (typically around 1/4th of the model depth) used to perform intermediate verification. It tentatively accepts draft tokens that align with its distribution and early-rejects inaccurate ones.
Target Layer ( $L_f$ ): The final layer (or full model) used for periodic full-model verification to ensure output consistency.

Key Technical Innovations

Low-Overhead Intermediate Verification: Unlike prior works that train separate auxiliary models, HiSpec uses the inherent exit layers of the EE-model. This eliminates the need for training a new verifier and avoids the memory overhead of loading multiple model weights.
KV Cache and Hidden State Reuse: To prevent redundant computation, HiSpec implements a mechanism to reuse Key-Value (KV) caches and hidden states across the draft generation, intermediate verification, and target verification stages. It buffers these states during draft generation and prunes entries associated with rejected tokens before the next phase, ensuring context alignment without recomputation.
Lossless Guarantee: HiSpec employs a two-stage verification process ( $L_d \to L_i \to L_f$ ). The intermediate verifier only performs early rejection; it does not relax the final acceptance criterion. By applying the lossless speculative decoding proof sequentially at both stages, HiSpec ensures the final output is identical to that of the target model running in isolation (auto-regressive decoding).
Dynamic Verification Policy: To balance throughput and accuracy, HiSpec accumulates a window of tentatively accepted tokens (default $N_i = 4$ ) before invoking the expensive full-model verification. This reduces the frequency of full-model calls while preventing long chains of unverified tokens.
Batched Inference Support: Unlike many prior speculative decoding methods that struggle with batching due to non-uniform token acceptance lengths, HiSpec integrates paged attention with block tables. This allows per-sequence acceptance without padding overhead, enabling efficient batched inference.

3. Key Contributions

The paper claims the following contributions:

Identification of the Verification Bottleneck: The authors demonstrate that verification latency (up to 6.9× slower than generation) is the primary limiter of speculative decoding throughput, a problem largely ignored by prior draft-acceleration works.
HiSpec Framework: A novel framework using Early-Exit models for low-overhead intermediate verification, eliminating the training and memory costs associated with auxiliary verifiers.
Resource Efficiency: Mechanisms to reuse KV caches and hidden states across all stages (draft, intermediate, target), significantly reducing compute and memory redundancy.
Accuracy Preservation: A design that maintains strict output consistency with the target model through periodic full-model verification, ensuring a lossless guarantee.
Generalizability and Scalability: The method works with both pre-trained EE-models and post-training modified models (via fine-tuning prediction heads) and natively supports batched inference.

4. Results

The authors evaluated HiSpec across eight state-of-the-art models (including Llama3, Llama2, CodeLlama, Qwen-3, and OPT) and five diverse benchmarks (ShareGPT, CNN/DM, HumanEval, GSM8K, XSum).

Throughput Improvement: Compared to baseline single-layer speculation (without intermediate verification), HiSpec improves throughput by 1.28× on average and up to 2.01×.
Comparison with SOTA: HiSpec consistently outperforms state-of-the-art draft-acceleration methods (AdaDecode, Lookahead, LayerSkip) and verification-acceleration methods (SPRINTER).
- Against SPRINTER (which uses an auxiliary verifier), HiSpec achieves 2.01× throughput on ShareGPT compared to SPRINTER's 1.20×, while maintaining 100% accuracy (SPRINTER degraded accuracy to ~40-52% in their tests).
- HiSpec outperforms LayerSkip and Lookahead across all benchmarks and model sizes.
Resource Efficiency: The intermediate verification adds negligible memory overhead (< 0.01%) due to KV cache reuse.
Batching: HiSpec maintains speedup across batch sizes (1 to 16), whereas prior methods often fail to scale effectively with batching.

5. Significance and Claims

The paper positions HiSpec as a solution to the critical "verification wall" in LLM inference. Its significance lies in:

Shifting the Focus: Moving the optimization focus from accelerating draft generation to accelerating verification, which the authors argue yields higher returns.
Efficiency: Achieving high throughput without the computational and memory costs of training auxiliary models or managing multiple model weights simultaneously.
Practicality: Being the first method to combine intermediate verification with native support for batched inference and lossless guarantees, making it suitable for widespread deployment.
Flexibility: The ability to adapt to both pre-trained EE-models and standard models modified via lightweight post-training (adding prediction heads), ensuring broad applicability.

The authors conclude that by exploiting early-exit layers for intermediate verification and efficiently managing resources, HiSpec provides a robust, high-throughput inference framework that does not compromise the accuracy of the target model.