TreeWY: Speculative Verification for Gated DeltaNet Hybrids

In the world of modern artificial intelligence, large language models act as powerful engines that generate text one word at a time. To do this, they must remember everything they have written so far to ensure the next word fits the context. For many years, the most common way to handle this memory was to keep a growing list of every previous word and its associated data, a method that works well but requires a massive amount of computer memory as the conversation gets longer. Recently, a new generation of models has emerged that mixes this traditional approach with a different, more compact method. These hybrid models use a special type of layer that summarizes the entire history into a single, fixed-size snapshot, rather than a growing list. This makes them incredibly efficient for standard reading and writing tasks, allowing them to run on smaller computers. However, this efficiency creates a new problem when trying to speed up the process using a technique called speculative decoding.

Speculative decoding is a strategy used to make these AI models faster. Instead of generating one word at a time, a smaller, cheaper "draft" model guesses several words ahead, and the main model checks them all at once to see which guesses were correct. If the main model can verify a long string of guesses, it can output multiple words in a single step, dramatically increasing speed. The problem arises with the hybrid models mentioned earlier. Because their memory is a single, fixed snapshot that cannot be easily cut or pasted, the current systems have to take a complete copy of this snapshot for every single word the draft model guesses. If the draft model guesses a long list of words, the system must store a copy of the memory for each guess. This quickly fills up the computer's memory, forcing the system to stop guessing long lists and limiting the speed gains. It is a bottleneck that prevents these efficient models from running as fast as they theoretically could.

Researchers at Thomson Reuters, led by Sneha Murthy Ghantasala, have developed a new method called TreeWY to solve this specific memory bottleneck. Their work focuses on a family of hybrid models known as Qwen3.5, which are designed to be efficient but struggle with the memory demands of speculative decoding. The team realized that the way these models update their memory follows a specific mathematical pattern that could be rearranged. Instead of saving a full copy of the memory state for every single guess, they found a way to calculate the results of all the guesses at once using a single, streamlined mathematical operation. This operation treats the sequence of guesses not as a series of separate snapshots, but as a structured tree where the necessary information is shared and computed efficiently.

The core of their discovery is that they no longer need to store the full memory state for every draft token. In the old method, if a model guessed ten words, it had to save ten full versions of its memory, which consumed a huge amount of space. With TreeWY, the system only saves a tiny, compressed summary of the changes made during the guessing phase. It performs a single calculation to verify all the guesses simultaneously. If a guess is accepted, the system reconstructs the correct memory state from that small summary. If a guess is rejected, the system simply discards the summary without ever having needed to store the full, heavy memory state in the first place. This approach removes the need for the massive memory overhead that previously made long guessing chains impossible.

The researchers tested this method on two sizes of the Qwen3.5 model, a 35-billion parameter version and a much larger 397-billion parameter version, running them on high-end graphics cards. They compared their new TreeWY method against the standard approach used in popular AI software. The results showed that when the computer's memory was under heavy pressure, the new method allowed the system to handle significantly more requests at once. In some cases, the speed of generating text increased by nearly fifty percent, and the time it took to start generating a response dropped dramatically. This happened because the freed-up memory space allowed the system to keep more active conversations running simultaneously without crashing or slowing down.

However, the study also found that the benefits depend heavily on how much memory is available. When the computer had plenty of memory to spare, the new method was slightly slower than the standard approach, costing a few percent in speed. This is because the new calculation, while saving memory, takes a tiny bit more time to perform than the simple method of just copying data. The researchers noted that the real win comes when the system is memory-limited, which is a common situation for large models. In these scenarios, the ability to run more conversations at once far outweighs the small cost of the calculation.

The team also explored whether this method could support even more ambitious guessing strategies, such as a "tree" structure where the model guesses multiple different paths at the same time, rather than just a single line of words. The new method made this possible by keeping the memory usage flat regardless of how wide the tree of guesses became. Previously, a wide tree of guesses would have required so much memory that it was impossible to run. Now, the system can afford to try many different paths at once. While this did not immediately result in a massive speed boost for the models tested, it proved that the method is flexible enough to handle complex, wide guessing structures that were previously unaffordable.

The researchers emphasized that their solution is specific to the mathematical rules governing these hybrid models. It does not rely on the specific details of the model's design but rather on the fundamental way the memory updates itself. This means the approach could potentially be applied to other models that use similar memory structures. The work was implemented in a widely used software framework for running AI models, and the researchers verified that the new method produces results that are mathematically identical to the standard approach, ensuring that the quality of the text remains unchanged.

In the end, this research demonstrates that by rethinking how memory is handled during the guessing phase, it is possible to unlock the full speed potential of efficient hybrid models. The study shows that the limitation was not in the models themselves, but in the way the software managed their memory. By switching from storing full snapshots to calculating a shared summary, the researchers turned a memory bottleneck into a pathway for higher performance. This allows these efficient models to run faster and handle more users, particularly in situations where computer memory is the tightest constraint. The findings suggest that future improvements in AI speed may come not just from building bigger models, but from smarter ways of managing the data they already hold.