💬 NLP

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

AsymSpec is an asymmetric speculative decoding framework that enables a lightweight drafter to access the full input context while a large verifier operates on a compressed view, effectively balancing inference speed and accuracy for agentic LLMs by achieving near-full-context performance at significantly reduced compute costs.

Original authors: Sheng Liang, Yongyue Zhang, Nathanael Brian, Hang Lv, Hao Wang, Chen Zhang, Yong Liu

Published 2026-08-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sheng Liang, Yongyue Zhang, Nathanael Brian, Hang Lv, Hao Wang, Chen Zhang, Yong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems are increasingly acting as autonomous agents, capable of retrieving documents, using software tools, and holding multi-turn conversations to solve complex problems. As these agents work, they accumulate a growing history of information: search results, tool outputs, and past messages. To make these systems fast enough for real-world use, engineers often compress this history, summarizing long documents into short sentences or stripping away detailed images to save time. However, this compression comes at a steep price: the AI loses the fine-grained details necessary for accurate reasoning, forcing a difficult choice between speed and intelligence. A standard technique called speculative decoding, which uses a small, fast model to guess what a large, slow model will say, has been the go-to solution for speeding up AI. Yet, this traditional method hits a wall in agentic settings because it requires both the small guesser and the large verifier to see the exact same information. If the information is compressed to save time, the guesser loses the same critical details as the verifier, meaning the speedup cannot recover the lost accuracy.

Researchers at Huawei Technologies and the University of Science and Technology of China have proposed a new approach called ASYMSPEC that breaks this deadlock by allowing the two models to see different versions of the same story. In their system, the large, powerful model that makes the final decision operates only on the compressed, fast version of the input to keep latency low. Meanwhile, a much smaller, lightweight model reads the full, uncompressed history in the background. This small model acts as a guide, identifying exactly what information was lost during compression and subtly nudging the large model's predictions to include those missing details. The researchers found that this asymmetric setup allows the system to retain nearly all the accuracy of a full-context analysis while operating at the speed of a compressed one. In tests across four different agentic capabilities, including complex multi-step questions and tool usage, the method recovered about 90 percent of the full-context accuracy while using only 20 to 30 percent of the computational cost.

The core innovation lies in how the two models communicate. Instead of simply letting the small model guess and hoping the large model agrees, the system compares the small model's reaction to the full text against its reaction to the compressed text. The difference between these two reactions reveals exactly what the compression removed. The system then uses this specific signal to adjust the large model's output, effectively filling in the gaps without requiring the large model to process the heavy, full-length data itself. A smart gatekeeper mechanism ensures this guidance is applied only when necessary, preventing the system from being confused when the compression is mild. This approach works even when the input involves different types of media; for instance, a small model can look at a raw image while the large model only sees a text description, with the small model steering the large one to understand the visual details it cannot see.

The researchers tested this method on real-world agent tasks, such as answering questions that require searching through multiple documents, following multi-turn instructions, and using software tools. They compared their system against standard methods that either process everything slowly or compress everything quickly. The results showed that traditional speculative decoding could not recover the accuracy lost to compression; it simply sped up the lossy process. In contrast, the new asymmetric method successfully bridged the gap, delivering performance close to the slow, full-context ideal but at a fraction of the time. On specific benchmarks involving long documents, the system achieved speedups of 1.3 to 1.7 times compared to the uncompressed baseline, while reducing the computing power required to roughly one-fifth. The study suggests that this technique is particularly valuable when the compression is severe, as the system's ability to recover lost information scales directly with how much detail was initially stripped away.

This work demonstrates that the trade-off between speed and accuracy in AI agents does not have to be rigid. By decoupling what the fast model sees from what the slow model sees, and using a lightweight guide to transfer critical insights, it is possible to have both efficiency and high-quality reasoning. The findings indicate that for tasks where context is heavy and details are easily lost, a small model reading the full picture can effectively steer a large model working with a summary. This approach offers a practical path forward for deploying sophisticated AI agents in production environments, where latency and cost are critical constraints, without sacrificing the reliability needed for complex decision-making.

Technical Summary: AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

1. Problem Statement

Modern agentic LLM pipelines (involving retrieval, tool use, and multi-turn interactions) face escalating inference costs as context accumulates. To manage latency, production systems routinely compress inputs (e.g., summarizing retrieved passages, using API signatures instead of full docs, or replacing raw images with captions). However, this compression systematically discards fine-grained details critical for task accuracy, creating a rigid trade-off: either accept prohibitive latency from full-context generation or suffer significant accuracy degradation.

Existing Speculative Decoding (SD) methods, which accelerate generation by using a lightweight drafter to propose tokens for a large verifier, cannot resolve this trade-off. Standard SD enforces a symmetric context constraint: both the drafter and the verifier must process identical input tokens. Consequently, if the verifier is compressed to save latency, the drafter is also forced to operate on the compressed view, inheriting the same accuracy loss. SD accelerates the compressed model but cannot recover the information discarded by compression.

2. Methodology: AsymSpec

The authors propose ASYMSPEC, a framework that breaks the symmetry of context access in speculative decoding. The core insight is a structural compute asymmetry: per-step latency is dominated by the large verifier, while the lightweight drafter adds negligible overhead. ASYMSPEC decouples context access to exploit this:

The Verifier: Operates strictly on the compressed view ( $x_{comp}$ ) to maintain low latency.
The Drafter: Reads the full input ( $x_{full}$ ) to reconstruct the information discarded by compression.

The framework recovers full-context reasoning fidelity through two coupled mechanisms:

2.1 Contrastive $\delta$ -Fusion

To steer the compressed verifier using the full-context drafter without requiring the verifier to attend to $x_{full}$ , ASYMSPEC employs a contrastive mechanism:

The drafter ( $S$ ) performs two forward passes: one on $x_{full}$ (producing logits $a$ ) and one on $x_{comp}$ (producing logits $b$ ).
The context-gain signal is isolated by subtracting the compressed logits from the full-context logits: $\delta = a - b$ .
This subtraction cancels the drafter's context-independent preferences, isolating the specific information gain provided by the uncompressed input.
The signal $\delta$ is fused into the verifier's logits ( $t$ ) via a weighted addition: $t' = t + \beta \delta$ , where $\beta$ controls steering strength.

2.2 Context-Divergence Acceptance (CDA) Gate

Standard SD uses a fixed acceptance threshold. However, when full and compressed views diverge sharply, a fixed threshold may over-reject drafts that contain critical recovered information. ASYMSPEC introduces a parameter-free Context-Divergence Acceptance (CDA) gate:

The effective threshold $\gamma_{eff}$ relaxes proportionally to the divergence between the full and compressed contexts.
Divergence ( $D_i$ ) is measured using the Jensen-Shannon Divergence (JSD) between the drafter's distributions on $x_{full}$ and $x_{comp}$ .
The threshold is modulated as $\gamma_{eff}(i) = \gamma \cdot \exp(-D_i)$ .
This ensures that positions with high context-induced divergence (where the compressed verifier is likely to under-score useful full-context drafts) are accepted more readily, while maintaining stability.

2.3 Cross-Modal Extension

The framework naturally extends to cross-modal settings. A vision-language drafter can process raw images ( $x_{full}$ ) while a text-only verifier operates on captions ( $x_{comp}$ ). Since the output token vocabulary is shared, the $\delta$ signal and acceptance logic remain well-defined, allowing the text verifier to be steered by visual information it cannot directly consume.

3. Key Contributions

Asymmetric Speculative Decoding Framework: ASYMSPEC introduces an operating point where a compressed verifier achieves near-ceiling accuracy by leveraging a full-context drafter, a configuration inaccessible to symmetric SD methods.
Dual Mechanisms: The paper instantiates this via a same-model cross-context $\delta$ -fusion (isolating context gain by canceling capacity biases) and a parameter-free CDA gate (bounding steering strength without per-dataset tuning).
Cross-Modal Capability: The framework enables text-only verifiers to perform vision-reasoning tasks by steering them with vision-language drafters, avoiding the need for multimodal infrastructure upgrades on the verifier side.

4. Experimental Results

Evaluated across four agentic capabilities (long-context multi-hop QA, multi-turn instruction following, tool use, multimodal reasoning) and two end-to-end benchmarks (GAIA, SimpleQA):

Accuracy Recovery: ASYMSPEC recovers approximately 90% of full-context accuracy on average. For instance, on LongBench multi-hop subsets, it closes 59–94% of the gap between the compressed "Floor" and full-context "Ceiling."
Efficiency: The method delivers 1.3–1.7× throughput speedups over full-context generation while operating at 0.2–0.3× the compute cost (FLOPs) on isolated text capabilities.
Compression Sensitivity: Accuracy gains scale monotonically with compression severity. On tasks with near-lossless compression (e.g., MultiChallenge), ASYMSPEC yields negligible improvement, confirming the mechanism activates specifically to recover discarded signals.
Cross-Modal Performance: On MathVista, ASYMSPEC outperforms symmetric SD by 10.1 points, recovering significant accuracy for visual question answering and figure question answering tasks where the text-only verifier otherwise fails.
Robustness: The method is robust across hyperparameters ( $\beta$ , $\gamma$ ) and compressor types (summarization, truncation, token pruning). It requires a drafter capacity of at least 1.7B parameters to reliably extract context-gain signals.

5. Significance and Claims

The paper claims that ASYMSPEC fundamentally changes the efficiency operating point for agentic LLMs. By breaking the symmetric context constraint, it allows systems to retain the low latency of compressed inference while recovering the reasoning fidelity of full-context generation.

The authors emphasize that this approach is particularly valuable for production agentic pipelines where context length is the primary driver of overhead. The method provides a predictable criterion for deployment: asymmetric steering is warranted precisely when compression discards critical reasoning signals. The paper concludes that ASYMSPEC offers a practical solution to the accuracy-overhead trade-off without requiring changes to the underlying verifier architecture or access to proprietary full-context data during the verification step.