💬 NLP

VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

The paper proposes VIA-SD, a multi-tier speculative decoding framework that leverages intra-model routing to deploy a slim submodel for verifying medium-confidence draft tokens, thereby reducing rejection rates and achieving significant inference speedups over existing methods without requiring training modifications.

Original authors: Yuchen Xian, Yang He, Yunqiu Xu, Yi Yang

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuchen Xian, Yang He, Yunqiu Xu, Yi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a long, complex story, but you have two editors: a Speedy Intern and a Master Editor.

In the world of Large Language Models (LLMs), the "Master Editor" is the giant, super-smart model that produces high-quality text but is very slow and expensive to run. The "Speedy Intern" is a tiny, fast model that can guess what comes next quickly, but it makes mistakes.

The Old Way: The "All-or-Nothing" Check

Traditionally, when the Intern guesses a sentence, the Master Editor has to check it.

If the Master Editor agrees, great! The Intern's guess is accepted.
If the Master Editor disagrees, the Intern's guess is thrown out, and the Master Editor has to write the whole sentence from scratch.

This is like a strict boss who says, "If you aren't 100% perfect, I'm going to do the whole task myself." This wastes a lot of time because the Master Editor often has to redo work that was actually almost right.

The New Idea: The "Middle Manager"

The paper introduces a new system called VIA-SD. It realizes that many of the Intern's guesses aren't terrible; they are just "okay" or "pretty good." They don't need the full power of the Master Editor to fix them.

So, VIA-SD adds a Middle Manager (called a "slim-verifier") between the Intern and the Master Editor. This Middle Manager is a special version of the Master Editor that is lighter and faster because it's built by "routing" (selecting specific parts) of the big model itself.

Here is how the new three-step process works:

The Easy Wins: The Intern guesses a word. If it's obviously correct, the system accepts it immediately. No one needs to check.
The "Good Enough" Zone: If the Intern's guess is a bit shaky, the Middle Manager steps in. Instead of rejecting the guess and asking the Master Editor to start over, the Middle Manager quickly fixes or rewrites the word itself. It's like a senior team member who can tweak a draft without needing the CEO's approval.
The Hard Cases: Only if the Middle Manager is truly confused does the system finally call in the Master Editor to do the heavy lifting.

The "Router" Trick

How do they build this Middle Manager? They don't train a whole new model from scratch. Instead, they take the giant Master Editor and turn off (skip) some of its internal layers, creating a "slim" version.

Think of the Master Editor as a massive factory with 50 assembly lines. The Middle Manager is that same factory, but with only 25 lines running. It's faster, but because it's built from the same blueprints and uses the same tools, it still understands the work perfectly. The paper uses a smart search method (called DIMR) to figure out exactly which 25 lines to keep to get the best balance of speed and accuracy.

Why This Matters

The paper claims that by adding this Middle Manager:

Fewer Rejections: The system rejects fewer guesses. Instead of throwing away "okay" work, it fixes it quickly.
Faster Speed: Because the giant Master Editor is called less often, the whole process becomes 10% to 20% faster than previous methods, and up to 3 times faster than standard decoding.
No Extra Training: This system works with existing models without needing to retrain them. It just changes how they verify answers during use.

In short, VIA-SD stops treating every mistake as a catastrophe. Instead of a binary "Pass or Fail" system, it creates a tiered system where "almost right" gets a quick fix, and only the truly broken parts get the full, slow attention of the giant model.

Technical Summary: VIA-SD

1. Problem Statement

Speculative Decoding (SD) is a widely adopted technique to reduce the inference latency of Large Language Models (LLMs). It operates on a two-tier "draft-verify" paradigm: a lightweight drafter model generates candidate tokens, which are then verified in parallel by a larger target model. If the verifier accepts the candidates, they are emitted; if rejected, the system falls back to full autoregressive decoding from the verifier.

The paper identifies a fundamental limitation in existing SD methods: they rely on a rigid binary decision rule (accept or fully recompute). The authors observe that many tokens fall into a "middle zone" where the drafter's prediction is close to the verifier's but not fully reliable. In current frameworks, these tokens trigger a full re-computation by the large model, incurring unnecessary computational costs. Furthermore, existing hierarchical approaches often rely on intermediate models to make binary decisions, failing to utilize intermediate models for active token generation or correction.

The core research question is whether middle-zone tokens can be handled by introducing a sequence of intermediate verification stages without increasing the distributional gap between the drafter and the verifier.

2. Methodology: VIA-SD

The authors propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework that replaces the binary accept-reject mechanism with a hierarchical verification process.

Theoretical Foundation: KL Geometry

The authors reframe SD from an information-theoretic perspective. While the Total Variation (TV) distance strictly governs rejection rates in lossless SD (implying intermediate steps cannot reduce rejection), the Kullback-Leibler (KL) divergence offers a more flexible geometric view.

Pythagorean Decomposition: Based on the generalized Pythagorean theorem in information geometry, the KL divergence between the drafter distribution $p$ and the verifier distribution $q$ can be decomposed via intermediate distributions $u$ : $D_{KL}(p \| q) \ge D_{KL}(p \| u) + D_{KL}(u \| q)$ .
Implication: This suggests that introducing an intermediate distribution $u$ can reduce the cumulative divergence (and thus the verification cost) compared to a direct $p \to q$ path, provided $u$ is a suitable projection.

Architecture: The Slim-Verifier

To operationalize the intermediate distribution $u$ , VIA-SD introduces a slim-verifier constructed via Dynamic Intra-Model Routing (DIMR).

Intra-Model Routing: Instead of training a separate intermediate model, the slim-verifier is a routed submodel derived from the full verifier $q$ . It is created by selectively skipping Transformer layers based on a routing mask $z$ .
Distributional Consistency: Because the slim-verifier shares embeddings and the output head with the full verifier, it maintains better distributional alignment than an independent intermediate model of the same size.
DIMR Optimization: The optimal routing mask is found offline using a combination of random search and Bayesian optimization to minimize a KL-style cost function, ensuring the slim-verifier is a stable approximation of the intermediate distribution.

The VIA-SD Pipeline

The inference process involves three stages of verification for a block of drafted tokens:

Direct Acceptance: High-confidence tokens (where the drafter $p$ aligns well with the slim-verifier $q'$ ) are accepted immediately.
Slim-Verifier Regeneration: Medium-confidence tokens (where $p$ is close but not perfect) are handled by the slim-verifier $q'$ . If $q'$ is confident, it rewrites the token; otherwise, it proceeds to the next stage.
Full-Model Verification: Low-confidence or uncertain tokens are escalated to the full verifier $q$ for final validation or re-generation.

This hierarchical gating allows the system to resolve "middle-zone" tokens using the computationally cheaper slim-verifier, reserving the full model only for the most difficult cases.

3. Key Contributions

Theoretical Reformulation: The paper introduces a KL-based information-theoretic view to justify the introduction of intermediate verification stages, moving beyond the limitations of the TV-distance-based binary paradigm.
Slim-Verifier Design: It designs a dynamically routed intra-model (slim-verifier) that approximates the large model's distribution efficiently, specifically targeting medium-confidence tokens to reduce redundant full-model calls.
VIA-SD Framework: A practical multi-tier framework is proposed and instantiated as a three-tier system (Drafter $\to$ Slim-Verifier $\to$ Full Verifier). This alleviates the binary limitation of traditional methods.
Compatibility: The method is compatible with existing SD frameworks (e.g., EAGLE, PEARL) without requiring retraining of the drafter or modification of the verifier's training procedure.

4. Experimental Results

The authors evaluated VIA-SD across four representative tasks (Summarization, Translation, Reasoning, QA, and Coding) using multiple model families (T5, Gemma2, LLaMA2, Qwen) and various size configurations (e.g., 2B $\to$ 9B, 7B $\to$ 70B).

Rejection Rate Reduction: VIA-SD consistently reduced rejection rates by 0.10–0.22 compared to strong speculative decoding baselines (e.g., Cascade SD, Faster Cascades).
Speedup: The method delivered 10–20% speedups over state-of-the-art speculative decoding baselines. Compared to non-drafting greedy decoding, it achieved 2.5–3 $\times$ acceleration.
Quality: Accuracy (measured by ROUGE, BLEU, and task-specific scores) was maintained or slightly improved, demonstrating that the "lossy" nature of the intermediate routing does not degrade generation quality.
Efficiency: The offline search cost for the routing mask (DIMR) was moderate (18–68 minutes per model pair) and incurred only once, while the online decoding speedups were significant.
Ablation: Experiments confirmed that the routed slim-verifier outperforms both independent intermediate models (which increase memory overhead) and random layer-skipping approaches (which fail to maintain distributional consistency).

5. Significance and Claims

The paper claims that VIA-SD establishes multi-tier speculative decoding as a general paradigm for scalable and efficient LLM inference.

Paradigm Shift: It moves the field from a binary "accept/reject" logic to a "staged allocation" of verification resources, effectively utilizing the "middle zone" of token uncertainty.
Practicality: By deriving the intermediate verifier from the target model itself (via layer skipping), the method avoids the memory and engineering overhead of loading separate intermediate models, making it highly practical for deployment.
Scalability: The results suggest that as the capacity gap between drafter and verifier increases, the benefits of hierarchical verification with a routed slim-verifier become more pronounced, offering a robust solution for accelerating large-scale LLM serving.