🤖 machine learning

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

ConfLayers is a training-free, adaptive layer-skipping method for self-speculative decoding that dynamically selects layers to skip based on confidence scores, achieving up to a 1.4x speedup over vanilla LLM generation while preserving output quality.

Original authors: Walaa Amer, Uday das, Fadi Kurdahi

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Walaa Amer, Uday das, Fadi Kurdahi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef (the Large Language Model) trying to write a complex recipe for a very picky food critic. The critic demands perfection, but you also need to serve the dish quickly because the restaurant is getting crowded.

Normally, to write every single word of the recipe, you have to think through every single step from scratch: chopping, sautéing, seasoning, plating. This is slow and tiring. This is how current AI models work: they generate text one word at a time, checking every single "layer" of their brain for every word.

ConfLayers is a new, clever way for this chef to speed things up without ruining the taste of the dish. Here is how it works, broken down into simple concepts:

1. The Problem: The "Double-Check" Bottleneck

In the old way of speeding up AI (called Speculative Decoding), the chef would hire a sous-chef (a smaller, faster model) to guess the next few ingredients. Then, the master chef would taste-test them to see if they were right.

The Catch: Hiring a sous-chef is expensive. You have to train them, store their recipes, and make sure they speak the same language as the master chef. Sometimes, the sous-chef guesses wrong so often that the master chef has to throw away the guess and start over, wasting time.

2. The Solution: "Self-Speculative" Cooking

ConfLayers says, "Why hire a sous-chef? Let the master chef be their own sous-chef!"
Instead of using a different model, the master chef uses a lighter version of themselves to make the guesses. They skip some of the heavy thinking steps (layers) to make a quick draft, and then use their full brain to verify it.

3. The Magic Trick: The "Confidence" Meter

Here is where ConfLayers gets really smart. Not all thinking steps are equally important.

Early steps (like "chop the onions") are very specific. If you skip these, the dish is ruined.
Later steps (like "add a pinch of salt") might be less critical if the chef is already very sure of the flavor.

ConfLayers gives the chef a Confidence Meter for every single step of the thinking process.

High Confidence: "I know exactly what word comes next. I don't need to double-check this step." -> Skip it! (Save time).
Low Confidence: "I'm not sure about this word. I need to think hard." -> Do the full work.

4. The "Adaptive" Filter (The Smart Window)

This is the paper's secret sauce. Imagine the chef looking at their confidence meter through a window.

If the chef is stable (confident), they look through a small window to check just the immediate next step.
If the chef is jittery (uncertain), they open a wide window to look at the whole picture to make sure they aren't making a mistake.

This "window" changes size automatically based on how the chef is feeling at that moment. It's not a rigid rule; it's a flexible, intuitive adjustment.

5. The "Practice Run" (Search Process)

Before the chef starts cooking the real meal for the critic, they do a quick, automated practice run.

They try skipping different layers.
They see how many guesses the full chef accepts.
They tweak the "skip list" until they find the perfect balance: Skip enough to be fast, but not so much that the food tastes bad.

Once they find this "Golden Recipe" for skipping, they use it for the rest of the night.

The Result: Faster, Not Cheaper

The paper shows that by using this method, the AI can generate text 1.4 times faster than usual.

Analogy: It's like driving a car. Instead of checking the speedometer, the road, the mirrors, and the engine every single second (slow), you only check the engine when you hear a weird noise (low confidence) and cruise normally when everything feels smooth (high confidence).

Why This Matters

No Extra Training: You don't need to teach the AI a new trick. It figures it out on the fly.
Works on Anything: It works on math problems, creative writing, and coding.
Quality Preserved: Because the "Master Chef" still verifies the final dish, the quality remains high. You don't get a faster, worse recipe; you get a faster, same-quality recipe.

In a nutshell: ConfLayers teaches AI to trust its gut feeling. When it's sure, it skips the boring details to save time. When it's unsure, it slows down and thinks hard. This makes AI faster without making it dumber.

1. Problem Statement

Large Language Models (LLMs) face significant computational and latency bottlenecks due to their autoregressive nature, where tokens are generated sequentially. While Speculative Decoding (using a draft model to propose tokens for a target model to verify) accelerates inference, traditional approaches require a separate, smaller draft model. This introduces:

Deployment Complexity: Need for additional training, parameter storage, and synchronization between two models.
Distribution Mismatch: Differences between the draft and target models can lower token acceptance rates, reducing speedup.
Limitations of Self-Speculative Decoding (SSD): Existing SSD methods (which reuse the same model for drafting and verifying) often rely on heuristic-based layer skipping (e.g., fixed rules) or policy-based learning (requiring fine-tuning). Heuristics lack adaptivity to diverse inputs, while learned policies incur high training costs and poor generalization.

The core challenge is designing a training-free, plug-and-play mechanism to construct adaptive draft subnetworks that balance decoding speed with high token acceptance rates without sacrificing output quality.

2. Methodology: ConfLayers

ConfLayers is a dynamic, confidence-driven framework that constructs draft models by selectively skipping transformer layers during inference. It operates in three main phases:

A. Confidence Computation

Instead of relying on external metrics, ConfLayers computes an internal confidence score for every intermediate layer during generation:

Logit Extraction: Hidden states from the Attention and Feed-Forward Neural Network (FFNN) modules of each layer are projected through the shared Language Model (LM) head to obtain logits.
Entropy Calculation: These logits are converted to probability distributions via Softmax. The Shannon Entropy ( $H(p)$ ) of this distribution is calculated to measure uncertainty.
Confidence Score: The confidence score is defined as the complement of normalized entropy:
$\text{confidence} = 1 - \frac{H(p)}{H_{\max}}$
- Scores near 1 indicate high confidence (sharp, peaked distribution).
- Scores near 0 indicate low confidence (uncertain, uniform distribution).

B. Adaptive Window Filtering

To determine which layers to skip, ConfLayers uses an adaptive windowing mechanism rather than a fixed threshold:

Standardization: Confidence values are standardized (zero mean, unit variance).
Gradient Analysis: The second-order discrete gradient of the confidence sequence is computed to measure local fluctuations.
Dynamic Window Size ( $w_i$ ): The size of the neighborhood window used to calculate the local mean and standard deviation adapts based on:
- Gradient Magnitude: High volatility (sharp changes) triggers larger windows for coarse-grained comparison; stable regions use smaller windows for fine-grained evaluation.
- Layer Depth: Early layers (lexical/syntactic) use smaller windows; later layers (contextual/abstract) use larger windows to capture long-range correlations.
Skipping Decision: A layer is skipped if its confidence falls below a local baseline:
$\hat{c}_i < \mu_{i,local} - \lambda \cdot \sigma_{i,local}$
where $\lambda$ is a sensitivity parameter tuned to maintain a skip ratio between 40% and 60%.

C. Iterative Search Process

ConfLayers employs a periodic search to refine the set of non-skip layers:

Initialization: Starts with a uniform layer skipping configuration.
Generation & Verification: The current draft model (subset of layers) generates candidate tokens, which are verified by the full target model.
Score Accumulation: The number of accepted tokens is accumulated over a defined search interval.
Update: If the average acceptance score improves or a maximum iteration limit is reached, the best-performing layer set is saved. The process repeats until no further improvement is found or a maximum number of iterations is reached.
Plug-and-Play: This search happens online during inference without retraining the model.

3. Key Contributions

Novel Confidence Mechanism: Introduces a method to estimate intermediate-layer confidence using predictive entropy, eliminating the need for auxiliary models or training.
Adaptive Filtering Algorithm: Designs an iterative, plug-and-play algorithm that dynamically adjusts layer skipping based on local confidence gradients and layer depth, balancing speed and token acceptance.
Comprehensive Evaluation: Demonstrates consistent acceleration and quality retention across diverse models (LLaMa-2/3, CodeLLaMa, Qwen) and tasks (summarization, math, translation, coding) compared to state-of-the-art baselines.

4. Experimental Results

The framework was evaluated on LLaMa-2 (13B, 70B), LLaMa-3 (8B, 70B), CodeLLaMa-34B, and Qwen2.5-Math-72B across datasets like CNN/Daily Mail, GSM8K, WMT14, and HumanEval.

Speedup: ConfLayers achieves up to 1.4× speedup over vanilla autoregressive generation.
- Example: LLaMa-3-70B on CNN-DM achieved 1.38× speedup, outperforming SWIFT (1.26×) and DEL (0.89×).
Comparison with Baselines:
- vs. DEL: DEL often fails to provide speedup (>1.0) because it requires fine-tuning for early exiting, whereas ConfLayers is training-free.
- vs. SWIFT: ConfLayers consistently outperforms SWIFT. While SWIFT optimizes for acceptance rate, ConfLayers optimizes for the total number of accepted tokens. This allows ConfLayers to use smaller draft models (skipping more layers) that still yield a higher absolute number of accepted tokens, resulting in better overall speedup.
Quality Retention: Rouge-2 scores indicate that output quality is preserved, as the method conserves the target model's probability distribution.
Dynamic Adaptability: ConfLayers maintains high performance (>1.1× speedup) even when a skip layer set optimized for one task (e.g., summarization) is applied to different tasks (e.g., math or translation), demonstrating robust generalization.
Ablation Study: Replacing the adaptive window with fixed window sizes (10 or 20) resulted in inconsistent or negative speedups (e.g., <0.9× on LLaMa-2-13B), proving the necessity of the adaptive filtering mechanism.

5. Significance

ConfLayers represents a significant step toward efficient, scalable LLM inference. By leveraging internal confidence signals to dynamically prune the computation graph, it offers a training-free, hardware-agnostic solution that avoids the overhead of managing separate draft models. The approach demonstrates that simple, confidence-guided heuristics can outperform complex learned policies, making high-speed speculative decoding more accessible for real-time interactive applications.