AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
AngelSpec introduces a unified training framework that co-specializes distinct drafting structures (MTP for chat and block-diffusion for code/math) and dynamically optimizes inference verification resources to achieve significant throughput improvements and higher acceptance rates across diverse real-world workloads.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a super-smart robot how to write stories, solve math problems, or write computer code. This robot, known as a Large Language Model (LLM), is incredibly talented, but it has a very specific quirk: it thinks one word at a time. To write a sentence, it must stop, think, and generate the first word, then stop again to think of the second, and so on. It's like a chef who can only chop one vegetable, then must wait for the entire kitchen to cool down before chopping the next. As the robot gets smarter and the requests get more complex, this "one-by-one" process becomes painfully slow, like watching paint dry in slow motion.
To speed things up, scientists invented a trick called "speculative decoding." Think of it as a team effort. You have a tiny, fast "drafting" robot that guesses the next few words in a sentence, and a giant, slow "target" robot that checks if those guesses are correct. If the big robot agrees, the team can write several words at once instead of just one. It's like a fast runner guessing the path ahead for a heavy walker; if the guess is right, they both move forward together, saving a massive amount of time. However, there's a catch: the fast runner isn't always right. If it guesses too many wrong words, the big robot has to waste time correcting them, and the whole process slows down. The big question for scientists is: how do we make the fast runner guess better, and how do we know when to stop guessing so we don't waste time?
This is where a new project called AngelSpec comes in. The researchers at Tencent realized that a single "fast runner" strategy doesn't work for every situation. Just like a sprinter might be great at a short race but terrible at a marathon, different types of text require different guessing strategies. For example, writing a casual chat message is chaotic and full of surprises (high entropy), while writing a math proof or a computer program is very structured and predictable. The paper argues that trying to use one "universal" guessing robot for everything is a mistake. Instead, they built a system that uses two different types of guessers depending on the task, and a smart manager that decides exactly how much time to spend on checking those guesses based on how busy the system is.
The team introduced a new method called DFly for the structured tasks like coding and math. Imagine DFly as a team of detectives who don't just guess the next clue one by one, but look at the whole puzzle at once to see the pattern. They use a special "hybrid" brain that looks at the big picture and then fine-tunes their guesses based on what came immediately before. This allows them to predict long chains of code or math steps with high accuracy. For the messy, creative chat tasks, they stick with a simpler, faster method called MTP (Multi-Token Prediction), which is great for quick, conversational bursts.
But having a great guesser isn't enough; you also need a smart manager. The paper introduces a feature called D-cut, which acts like a traffic controller. In a busy server, sometimes the "checking" process gets backed up. D-cut looks at the confidence of the guesses in real-time. If a guess looks shaky, D-cut cuts it off early to save time. If the system is running smoothly and the guesses look solid, it lets the team keep going longer. It's like a conductor who speeds up the orchestra when the music is easy and slows it down when the notes get tricky, ensuring the whole performance stays fast without crashing.
The results of this approach are impressive. When they tested their new system on the Hy3-A21B model, they found that it could process text much faster than before. Specifically, the system achieved a speedup of 1.98 to 2.40 times compared to the old "one-word-at-a-time" method. Even better, it was 10.5% to 11.8% faster than the previous best method known as DFlash. The researchers showed that by tailoring the guessing strategy to the specific type of text (chat vs. code) and using the smart manager to adjust to the workload, they could get the most out of their computing power. They even released their entire toolkit, called AngelSpec, to the public so other scientists can use these tricks to make their own AI models faster and more efficient.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.