How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
This paper demonstrates that while Orthrus achieves lossless speculative decoding with exact trajectory matching in FP32, its claimed losslessness degrades significantly under BF16 precision, highlighting that numerical precision critically impacts trajectory equivalence even when downstream task performance remains unaffected.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern computers that write text, known as language models, have become a dominant force in how we generate information. These systems work by predicting the next word in a sentence based on the words that came before it, one step at a time. This method is reliable but inherently slow, because the computer must finish calculating the first word before it can even begin to think about the second. As these models grow larger and the conversations get longer, this step-by-step process becomes a bottleneck, making the technology expensive and sluggish to use. To speed things up, researchers have developed a technique called speculative decoding. This approach tries to guess several future words at once, like a reader skimming ahead in a book, and then checks if those guesses are correct. If the guesses are right, the computer saves a tremendous amount of time.
One recent system, called Orthrus, promised to do this without losing any accuracy. It combines a standard, slow text generator with a faster, parallel guessing engine. The creators claimed that a built-in checking mechanism would ensure the fast engine produced the exact same sequence of words as the slow, original engine, making the acceleration truly "lossless." Lossless, in this context, means that the final output is identical to what the original, slower model would have produced, down to the very last character. This promise was significant because it suggested we could have the best of both worlds: the speed of parallel guessing with the perfect reliability of the original model.
A team of researchers decided to test this promise independently. They built their own version of the Orthrus system and compared its output against the original model across a wide variety of tasks, including writing code, solving math problems, and composing poetry. They ran these tests using a standard level of numerical precision that most modern computers use for efficiency. When they compared the word sequences generated by the fast Orthrus system against the slow original system, they found a surprising result. The two systems did not always agree. In fact, for the original released model, the sequences matched perfectly only about 45 percent of the time. For the researchers' own independently trained version, the match rate was even lower, at 43 percent. This means that in more than half of the cases, the fast system took a slightly different path, choosing different words than the original model would have.
The researchers dug deeper to understand why this happened. They discovered that the likelihood of the systems agreeing was connected to how difficult the text was for the original model to predict. When the original model was very confident in its next word, the fast system usually matched it. However, when the text was more complex or the model was less certain, the fast system was more likely to diverge and choose a different word. This suggests that the "lossless" claim was not holding up under the specific conditions of standard computer calculations. The researchers also noted that this divergence did not necessarily make the text worse. When they tested the models on standard benchmarks for reasoning and coding, the fast system sometimes scored slightly higher than the slow one, showing that a different path does not always mean a poorer result.
To solve the mystery of why the systems disagreed, the researchers changed the way the computer handled numbers. They repeated the entire experiment using a higher level of numerical precision, which allows the computer to store numbers with much greater exactness. When they made this switch, the result changed completely. Under this more precise calculation, the fast Orthrus system matched the slow original model perfectly every single time, across all 1,190 test prompts. This finding revealed that the earlier disagreements were not caused by a flaw in the design of the Orthrus system itself, but by the tiny rounding errors that happen when computers use standard, less precise math.
The study concludes that the promise of a "lossless" acceleration depends entirely on the mathematical precision used by the computer. While the Orthrus system works as intended in theory, the practical reality of running it on standard hardware introduces small errors that can change the final output. The researchers argue that when scientists claim a system is lossless, they must specify the level of numerical precision being used, because a system that is perfectly accurate in one setting may produce different results in another. This work clarifies that while we can make language models much faster, ensuring they produce the exact same output as the original requires careful attention to the underlying math, not just the architecture of the model.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.