💬 NLP

X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding

This paper proposes X-CoSD, a communication-efficient and lossless collaborative speculative decoding framework that enables distributed LLM inference between devices and servers with heterogeneous vocabularies by introducing hybrid resampling and a server-resampling-with-device-verification variant to significantly accelerate token generation without compromising output quality.

Original authors: Jaeduk Lee, Wan Choi

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaeduk Lee, Wan Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of modern artificial intelligence, the most powerful tools are large language models, vast digital brains capable of writing code, composing stories, and solving complex problems. However, these models are heavy and slow to run, often requiring massive servers far away from the user. To make them faster, researchers have developed a technique called speculative decoding. Imagine a team where a quick, lightweight assistant drafts a few sentences, and a senior expert checks them instantly. If the assistant is right, the team moves forward quickly; if the assistant is wrong, the expert corrects the mistake. This collaboration allows the system to generate text much faster than the expert could work alone. Yet, for this teamwork to happen between a phone and a distant server, they must speak the exact same language, down to the specific list of words and symbols they recognize. In the real world, different devices often use different dictionaries, creating a barrier that has previously stopped this fast collaboration from working smoothly.

A team of researchers at Seoul National University has solved this barrier with a new framework they call X-CoSD. Their work addresses a specific bottleneck: when the phone and the server do not share the same vocabulary, the server must send a massive amount of data back to the phone every time a mistake is made, slowing the entire process to a crawl. The researchers designed a system that allows the phone and server to collaborate even when their dictionaries do not match, without sacrificing the quality of the writing or the speed of the response. They proved that their method preserves the exact intelligence of the powerful server model while drastically cutting down the data that needs to travel over the internet.

The core of the problem lies in how these models handle errors. When the server checks the phone's draft and rejects a token, it must provide a new, correct option. In previous attempts to fix the vocabulary mismatch, the server would send its entire probability map—a list of every possible word it could choose next—back to the phone. This is like a teacher sending a student the entire textbook every time the student gets a single spelling wrong. It is incredibly inefficient, especially on wireless networks where sending large amounts of data takes time and power. The researchers realized that the phone only needs to see the words that both it and the server agree on, plus a tiny bit of extra information to handle the words only the server knows.

To solve this, the team introduced a method called hybrid resampling. Instead of sending the full dictionary of possibilities, the server sends only the probabilities for the words that appear in both the phone's and the server's vocabularies. It also sends a single number representing how much weight the server's unique words carry. With this limited information, the phone can mathematically decide whether to pick a word from the shared list or ask the server to pick a word from its unique list. If the phone picks a word from the shared list, it happens instantly. If the server needs to pick a unique word, it sends just that one word back, rather than the entire list of options. This approach ensures that the final output remains perfectly accurate, matching what the powerful server would have produced on its own, but it avoids the heavy data transfer that used to bog down the system.

The researchers went a step further with an enhanced version called X-CoSD-E, which adds another layer of efficiency. In this setup, when a mistake occurs, the server first sends a small handful of replacement candidates to the phone. The phone checks these few options immediately. If one of them is good enough, the process stops there, and no further data is sent. The server only sends the larger list of probabilities if the phone rejects every single one of the initial candidates. This "try a few first" strategy means that in most cases, the system avoids the heavy data transfer entirely. The team tested this on various tasks, including machine translation, summarizing news articles, and solving math problems, using different models for the phone and the server.

The results showed that this new method works as well as the powerful server model working alone, maintaining the same high quality of text generation. At the same time, it significantly reduced the amount of data sent back and forth. In their experiments, the new system generated tokens much faster than previous methods that tried to handle different vocabularies, particularly when the internet connection was slow or the data transfer was limited. The researchers found that by carefully managing what information is sent and when, they could keep the collaboration seamless. This work demonstrates that high-speed, collaborative artificial intelligence is possible even when the devices involved do not speak the exact same language, opening the door for more efficient and accessible AI tools on everyday devices.

Problem Statement

Collaborative Speculative Decoding (CoSD) is a distributed inference framework where an on-device Small Language Model (SLM) drafts candidate tokens, and a server-side Large Language Model (LLM) verifies them. While CoSD accelerates generation and offloads computation, existing methods rely on a critical assumption: the SLM and LLM must share the same vocabulary.

In practical heterogeneous settings (e.g., different model families or vendors), vocabularies often differ. A straightforward extension of CoSD to these settings requires transmitting the full server LLM probability distribution over the entire vocabulary whenever a token is rejected, so the device can perform exact resampling. This creates a prohibitive communication bottleneck, particularly on the downlink, which dominates end-to-end latency in wireless networks. Furthermore, existing communication-efficient CoSD methods (like U-HLM) often sacrifice decoding quality (lossy) or still require heavy uplink transmission of distributions.

Methodology

The authors propose X-CoSD, a framework designed for heterogeneous SLM–LLM vocabularies that guarantees lossless decoding (preserving the server LLM distribution) while minimizing communication overhead. The framework relies on two core mechanisms:

1. Hybrid Resampling (HR)

X-CoSD builds upon the Token-Level Intersection (TLI) approach, where the on-device SLM drafts tokens only from the common vocabulary ( $V_C$ ) shared by both models. When a token is rejected, the residual distribution (the probability mass needed for resampling) is decomposed into two regions:

Common Vocabulary Region ( $V_C$ ): Tokens present in both models.
LLM-Only Region ( $V_O$ ): Tokens present only in the server LLM.

Instead of sending the full distribution, HR exploits the structure of the residual distribution to perform distributed resampling:

The server calculates the total residual probability mass for $V_C$ ( $\theta_c$ ) and $V_O$ ( $\theta_o$ ).
The server transmits only the LLM probabilities for the common region ( $V_C$ ) and the scalar $\theta_o$ to the device.
The device probabilistically selects a resampling region:
- If $V_C$ is selected, the device locally resamples from the common region.
- If $V_O$ is selected, the device requests the server to sample from the LLM-only region. The server then sends only the detokenized string (not the full distribution) corresponding to the sampled token.

This approach ensures that the full LLM distribution is never transmitted, significantly reducing downlink load.

2. Server Resampling with Device Verification (SR-DV)

To further reduce overhead, the authors propose X-CoSD-E, an enhanced variant using SR-DV.

Upon rejection, the server samples a small number of replacement candidates ( $K$ ) from the full LLM distribution and sends only these tokens and their probabilities to the device.
The device verifies these candidates using a specific acceptance probability function ( $\alpha(x)$ ) derived in the paper.
If a candidate is accepted, the process ends. If all $K$ candidates are rejected (or after $M$ iterations), the system falls back to HR to ensure exact residual resampling.
This probabilistic approach avoids transmitting the common-region distribution in most cases, further lowering latency.

Key Contributions

Cross-Vocabulary Support: X-CoSD is the first CoSD framework explicitly designed for heterogeneous vocabularies without requiring model retraining or online knowledge distillation.
Lossless Guarantee: The paper provides theoretical proofs (Theorems 1 and 2) demonstrating that both X-CoSD and X-CoSD-E preserve the exact server LLM distribution, ensuring generation quality identical to the server LLM.
Communication Efficiency: By restricting distribution transmission to the common vocabulary region (HR) or replacing it with candidate tokens (SR-DV), the framework drastically reduces downlink communication load compared to straightforward extensions.
SR-DV Mechanism: The introduction of a fallback mechanism that prioritizes low-latency candidate exchange over full distribution transmission.

Experimental Results

The authors evaluated X-CoSD and X-CoSD-E using Vicuna-68M (SLM) with Llama-3.1-8B and Qwen2-7B (LLMs) across tasks including machine translation (WMT-DeEn), summarization (XSum, CNN/DailyMail), math reasoning (GSM8K), and QA (MMLU).

Generation Quality: Both X-CoSD and X-CoSD-E achieved generation quality scores comparable to the server LLM (relative scores $\approx$ 1.0), confirming the lossless property. In contrast, lossy baselines (Greedy/Target Resampling) and U-HLM showed significant quality degradation.
Communication Load:
- Downlink: X-CoSD reduced downlink load by orders of magnitude compared to the "Straightforward" extension (which sends the full distribution). X-CoSD-E further reduced this load by avoiding distribution transmission in many rejection cases.
- Uplink: Unlike some baselines that require uploading SLM distributions, X-CoSD maintains low uplink load, transmitting only token IDs and probabilities.
Throughput and Latency:
- X-CoSD and X-CoSD-E achieved higher token throughput than the server LLM alone and outperformed the Straightforward baseline, particularly as downlink bandwidth increased.
- X-CoSD-E demonstrated the lowest per-token latency among lossless methods, maintaining high throughput even under varying downlink transmission rates (100–300 Mbps).

Significance

The paper claims that X-CoSD and X-CoSD-E represent a practical step toward real-world deployment of collaborative LLM inference. By addressing the vocabulary mismatch problem without sacrificing decoding accuracy, these frameworks enable efficient, lossless collaboration between diverse on-device models and powerful server LLMs. This is particularly significant for wireless environments where communication bandwidth is a primary bottleneck, allowing for faster inference without compromising the quality of the generated text.