Efficient Block-Layer Parallel Inference for Vision-Language-Action on Hybrid Architectures
This paper presents a hybrid CPU-GPU inference framework that partitions Vision-Language-Action (VLA) models at the block-layer to offload computation to the CPU via an asynchronous pipeline, significantly reducing inference latency and GPU memory usage to enable successful deployment on resource-constrained autonomous driving platforms.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Autonomous driving has long relied on a modular approach, where separate computer programs handle distinct tasks like seeing the road, predicting where other cars will go, and deciding how to steer. These systems work together, but they are often rigid and struggle to adapt to complex, changing situations. Recently, a new type of artificial intelligence called a Vision-Language-Action model has emerged as a promising alternative. These models act more like a human driver who can look at a scene, understand a spoken instruction or a complex traffic scenario, and immediately decide on a physical action, all within a single, unified brain. While these models show great potential for creating smoother and safer driving, they come with a significant hurdle: they are incredibly heavy on computer resources. They demand massive amounts of processing power and memory, often overwhelming the specialized graphics chips found in current self-driving cars. This creates a bottleneck where the car's computer cannot keep up with the model's needs, making it difficult to install these advanced systems on real vehicles without sacrificing speed or safety.
Researchers at City University of Hong Kong and the Mohamed bin Zayed University of Artificial Intelligence have developed a new way to run these powerful models that solves this resource problem without changing the models themselves. Instead of forcing the entire artificial intelligence brain to run on the graphics chip, they split the work between the graphics chip and the car's main central processor. They found that while the graphics chip is excellent at the initial task of "seeing" the road, the central processor, which usually sits idle during these heavy visual tasks, is perfectly capable of handling the later stages of reasoning. By dividing the model into layers and assigning the first layers to the graphics chip and the final layers to the central processor, they created a balanced system. Furthermore, because a car is constantly moving and receiving a stream of new images, the researchers designed a pipeline where the central processor works on the reasoning for the current moment while the graphics chip is already starting to process the next moment. This overlapping of tasks allows the system to keep both processors busy, significantly speeding up the decision-making process.
When the team tested this approach on two leading autonomous driving models, the results were substantial. For one model named Orion, the time it took to make a driving decision dropped from 521 milliseconds to 408 milliseconds, a reduction of nearly 22 percent. For another model called MindDrive, the time fell from 443 milliseconds to 306 milliseconds, a decrease of over 30 percent. Perhaps even more critical than speed was the relief placed on the car's memory. The original Orion model required 45 gigabytes of memory on the graphics chip, a demand so high that it could not run alongside other essential car systems on the test vehicle. The new hybrid approach reduced this requirement to 29 gigabytes, allowing the model to run successfully on the same hardware where the original version failed completely. Throughout these tests, the quality of the driving decisions remained unchanged; the car did not drive worse or crash more often, proving that the speed gains did not come at the cost of safety.
The researchers also discovered that simply splitting the work was not enough; the timing of that split mattered greatly. If they assigned too much work to the central processor, it became the new bottleneck, slowing everything down. If they assigned too little, the graphics chip remained overloaded. They found a specific balance point where the workload was evenly distributed, allowing the system to achieve its fastest speed. They also confirmed that the overlapping pipeline was essential. Without it, the system would still have to wait for one task to finish completely before starting the next, negating the benefits of using two processors. By letting the processors work on different frames of video simultaneously, the system achieved a steady, high-speed flow of decisions.
This work demonstrates that the limitations of current self-driving hardware may not be a dead end, but rather a signal to rethink how tasks are distributed. The study suggests that by treating the car's computer as a team of specialists rather than a single powerhouse, it is possible to deploy advanced, large-scale artificial intelligence models on existing vehicles. The researchers validated this by running their system on a real vehicle alongside a standard suite of driving software, showing that the hybrid approach is not just a theoretical improvement but a practical solution that can coexist with the complex software already required to drive a car. The findings indicate that efficient system design, rather than just waiting for more powerful hardware, is the key to bringing these sophisticated driving models into the real world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.