Accelerating Visual On-Policy Distillation with Batched Speculative Jacobi Rollouts
This paper introduces HB-SJD, a batched Speculative Jacobi Decoding backend that accelerates visual on-policy distillation by enabling parallel multi-token processing and adaptive execution modes, significantly reducing training time while maintaining generation quality without altering the distillation framework.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, there is a constant tug-of-war between creating models that are smart enough to produce stunning images and keeping them small enough to run quickly. Large systems, often called teachers, can generate incredibly detailed pictures by predicting one tiny piece of the image at a time, building the whole picture from a single starting point. However, this step-by-step process is slow and expensive. To make these systems faster, researchers use a technique called knowledge distillation, where a smaller, simpler student model learns to mimic the teacher. A newer, more effective version of this process, known as on-policy distillation, asks the student to practice by generating its own training examples in real-time. This ensures the student learns exactly how it will behave when it is actually creating art, rather than just memorizing old examples. The problem is that this practice session is painfully slow; the student must still build its images one tiny piece at a time, which creates a bottleneck that slows down the entire learning process.
A team of researchers from Shenzhen has found a way to break this bottleneck without changing the fundamental rules of how the student learns. They introduced a new method called HB-SJD, which acts as a faster engine for the student's practice sessions. Instead of forcing the student to build an image one token, or tiny visual unit, at a time, this new system allows the model to propose several future pieces of the image simultaneously and then check if those guesses are correct. Think of it like a student taking a test: instead of writing one answer, waiting for a teacher to grade it, and then writing the next, the student writes a whole page of answers at once, and the teacher quickly marks which ones are right. The student then keeps the correct answers and only rewrites the ones that were wrong, moving forward much faster.
The researchers discovered that simply running this faster method on a group of images at the same time was not enough. In a standard setup, if one image finishes its generation quickly, it would have to wait for the slowest image in the group before the system could move on to the next step. This waiting period wasted valuable computing power. To solve this, the team designed their system so that each image could progress at its own speed. If an image is finished, it steps aside, and the system continues working on the remaining images without pausing. Furthermore, they found that the best way to run the system changes as the number of active images shrinks. When many images are being processed, the system works best by keeping the full group active. But as images finish and the group gets smaller, the system switches to a leaner mode that only processes the remaining active images, avoiding the overhead of managing empty slots.
When tested on image generation models, this new approach proved to be highly effective. The researchers found that it could cut the time required for the student to generate its practice images by nearly half, with speedups ranging from 1.4 to 1.6 times faster than the traditional method. Crucially, this speed did not come at the cost of quality. The images produced by the student models trained with this new method were just as sharp and realistic as those trained with the slower, old method. The system worked consistently well whether the images were short or long, whether the group of images being processed was small or large, and at different stages of the training process. By replacing only the engine that generates the practice data, the researchers managed to make the entire learning process significantly faster, proving that you can accelerate the training of artificial intelligence without compromising the quality of the final result.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.