💻 computer science

Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference

Jetson-PI is a novel framework for deploying Vision-Language-Action models on low-power onboard devices like the Jetson Orin, which achieves real-time control by combining a foresight-aligned future correction module to resolve perception-execution misalignment with confidence-based scheduling and system-level optimizations to minimize reaction time and latency.

Original authors: Zebin Yang, Qi Wang, Yunhe Wang, Xiurui Guo, Bo Yu, Shaoshan Liu, Jiafeng Xu, Hao Dong, Meng Li

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zebin Yang, Qi Wang, Yunhe Wang, Xiurui Guo, Bo Yu, Shaoshan Liu, Jiafeng Xu, Hao Dong, Meng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to fold a shirt or pick up a bowl. You give it a brain (a super-smart AI model) that looks at a picture and a command like "pick up the black bowl," then tells the robot's arm what to do. This is called a Vision-Language-Action (VLA) model.

The problem? These brains are huge and hungry. If you try to run them on a robot's own onboard computer (like a Jetson Orin, which is powerful but tiny compared to a desktop), the robot gets stuck in "thinking mode." It takes so long to calculate the next move that by the time the robot actually moves, the world has already changed. It's like trying to catch a ball while wearing glasses that show you where the ball was a second ago, not where it is now. The robot misses.

The Big Idea: "Foresight" and "Async"
The authors of this paper, Jetson-PI, came up with a clever way to make these robots think faster and move smarter without needing a massive, power-hungry supercomputer. They call their method Jetson-PI.

Here is how they fixed the two main headaches:

1. The "Time Travel" Fix (Solving Misalignment)

The Problem: In the old way, the robot looks at a picture, thinks for a long time (let's say 1.4 seconds), and then moves. But in those 1.4 seconds, the robot might have already bumped into something, or the object might have shifted. The robot is acting on "old news."

The Solution: Instead of just waiting, Jetson-PI uses a "Foresight-Aligned" trick. Imagine the robot has a tiny, super-fast crystal ball (a lightweight "future correction module").

The robot commits to a move (like "reach for the bowl").
The crystal ball instantly predicts what the world will look like after that move is finished.
The robot then uses this "future view" to plan the next move.

It's like a chess player who doesn't just look at the board now, but instantly visualizes the board after their opponent's next move, allowing them to plan their own next move perfectly. This stops the robot from acting on outdated information.

2. The "Smart Skip" Fix (Solving Slow Reactions)

The Problem: Even with the crystal ball, the robot still needs to check the real world occasionally. But checking the world is slow because the "big brain" (the VLM) is heavy. If you check the world every single time, the robot moves too slowly.

The Solution: The authors introduced a "Confidence-Based Scheduler."

The crystal ball (future module) also has a "confidence meter." It says, "I'm 90% sure I know what's happening next!"
If the confidence is high, the robot skips the heavy brain check and just uses the crystal ball to keep moving. It's like driving on a familiar road where you don't need to check the GPS every second.
If the confidence drops (maybe the robot is picking up a tricky object), the scheduler says, "Okay, pause! Let's check the real world with the big brain to be safe."

This means the robot only does the heavy lifting when it absolutely has to, making it much snappier.

3. The "System Upgrade" (Speeding Up the Engine)

The authors also realized that the software running on the robot was being inefficient. They treated the robot's computer like a busy kitchen:

No Rebuilding the Oven: Instead of rebuilding the cooking instructions (the computation graph) every time, they built it once and reused it.
No Walking to the Fridge: They kept all the ingredients (data) right on the counter (GPU memory) so the chef didn't have to walk back and forth to the fridge (CPU memory).
Unrolling the Steps: They combined many small steps into one big, smooth motion.

The Results: Does it Work?

The authors tested this in simulations and on real robots.

Speed: On a Jetson Orin, their method made the robot control frequency 8.66 times faster than using standard PyTorch and 5.41 times faster than using a tool called vla.cpp.
Success: In a test called LIBERO (a benchmark for robot tasks), their robot succeeded 14.8% more often than a previous top method called VLASH.
Battery: They showed that using a powerful desktop card (RTX 4090) would drain a robot's battery 6.0 times faster than using the onboard Jetson Orin. Jetson-PI lets the robot run on its own battery for much longer.

What They Explicitly Say Doesn't Work
The authors tested a few ideas and ruled them out:

Parallel Processing: They tried running the "big brain" and the "action expert" at the exact same time to save time. They found this fails on small onboard computers because they both fight for the same limited data bandwidth, slowing everything down. It only works on huge desktop computers.
Just Predicting Robot State: Previous methods tried to guess where the robot's arm would be in the future. The authors found this isn't enough because it doesn't account for how the environment (the table, the objects) changes. You need to predict the environment, not just the robot.

How Sure Are They?
The paper is very confident in the numbers they measured. They ran extensive simulations on the LIBERO benchmark and tested on real robots (like the X2-W) in a lab setting. They measured the exact milliseconds of delay and the exact success rates. They state clearly that while their method is a huge improvement for mobile robots, it still can't fully match the raw power of a massive cluster of supercomputers if the models get even bigger in the future. But for a robot that needs to move around on its own battery? They suggest this is a practical, working solution.

In short, Jetson-PI teaches a robot to "think ahead" and "skip the heavy lifting" when it's safe, allowing it to move fast and smart without needing a supercomputer in its backpack.

Technical Summary: Jetson-PI

Problem Statement

Vision-Language-Action (VLA) models have demonstrated remarkable performance in embodied tasks by leveraging pre-trained Vision-Language Models (VLMs) and fine-tuning on robot demonstration datasets. However, deploying these models on low-power onboard devices (e.g., NVIDIA Jetson Orin) faces two critical barriers:

High Inference Latency: The computational complexity of VLA models results in substantial inference latency (e.g., ~~1.4s on Jetson Orin), leading to low control frequencies (~~0.7 Hz) and slow reaction times.
Limitations of Existing Asynchronous Inference: While asynchronous inference (parallelizing action execution with the next inference) can hide latency, it introduces two specific issues on edge devices:
- Perception-Execution Misalignment: By the time an action chunk is predicted and executed, the environment has evolved relative to the input image used for prediction. This misalignment worsens with longer inference times, causing prediction accuracy to drop.
- Long Reaction Time: The robot's reaction speed remains bounded by the VLA inference time. Furthermore, existing asynchronous methods often require large action chunks to maintain stability, which further increases the average reaction time to environmental changes.

Deploying VLA models on high-end GPUs (e.g., RTX 4090) is not a viable solution for mobile robotics due to severe power consumption issues (reducing battery life by 6.0×) and the need for network connectivity if run as servers.

Methodology: Jetson-PI

The authors propose Jetson-PI, a framework for efficient VLA deployment on onboard devices via Foresight-Aligned Asynchronous Correction. The method addresses the identified challenges through three core components:

1. Foresight-Aligned Asynchronous Correction

To solve the perception-execution misalignment, the authors introduce a lightweight Future Correction Module.

Mechanism: Instead of predicting future images or robot states alone, this module predicts the future compressed VLM representation (latent state) at time $t + \Delta$ (where $\Delta$ is the inference latency), conditioned on the actions committed for execution.
Training: A two-stage training pipeline is used. First, the action expert learns to predict actions from future hidden states. Second, the correction module is trained to predict these future states from the current state and committed actions, outputting a confidence estimate.
Benefit: This allows the action expert to predict actions based on the future environment state rather than the outdated observation, aligning the prediction with the execution time. The module is extremely lightweight (40M parameters, ~1% of the total VLA size).

2. Confidence-Based Scheduling Optimization

To reduce reaction time, the authors propose a dynamic scheduling strategy that balances VLM and Action Expert invocations.

Logic: The Future Correction Module acts as a scheduler. If its predicted confidence is above a threshold $\theta$ , the system skips the heavy VLM inference and relies solely on the Action Expert to generate subsequent action chunks. If confidence drops (indicating high uncertainty or significant environmental change), the VLM is invoked to update the KV cache and hidden states.
Rationale: This avoids the "naive" approach of running VLM and Action Expert in parallel, which the authors show is inefficient on onboard devices due to bandwidth contention (both components become bandwidth-bound on edge hardware, unlike on high-end GPUs where VLM is compute-bound).

3. System-Level Accelerations

The authors built an inference framework based on llama.cpp tailored for VLA characteristics on edge platforms:

CUDA Graph Reuse: Since VLA token lengths are deterministic (fixed image resolution and short instructions), the CUDA graph is constructed once and reused, eliminating graph construction overhead.
GPU-Resident Intermediate Buffering: Intermediate results between components (ViT $\to$ LLM $\to$ Action Expert) are kept in GPU memory to avoid CPU-GPU communication latency.
Flow Matching Unrolling: The iterative denoising steps of the Action Expert are unrolled into a single unified computation graph, reducing the number of kernel launch calls.

Key Contributions

Identification of Edge-Specific Challenges: The paper highlights that standard asynchronous inference fails on edge devices due to bandwidth contention and the specific nature of perception-execution misalignment, which cannot be solved by predicting robot states alone.
Foresight-Aligned Correction: A novel module that predicts future VLM latents conditioned on committed actions, enabling the action expert to operate on future environmental representations.
Adaptive Scheduling: A confidence-driven mechanism that dynamically adjusts the frequency of VLM invocations, significantly reducing reaction time without sacrificing accuracy.
Edge-Optimized Inference Engine: A system design leveraging graph reuse, memory residency, and unrolling to maximize throughput on bandwidth-constrained devices.

Experimental Results

The authors evaluated Jetson-PI on the LIBERO benchmark (using the $\pi0.5$ model) and on real-world robots (PrimeBot X2-W).

Control Frequency: On NVIDIA Jetson Orin, Jetson-PI achieves an 8.66× improvement in control frequency compared to naive PyTorch inference and a 5.41× improvement compared to vla.cpp.
Success Rate: On the LIBERO benchmark, Jetson-PI outperforms the state-of-the-art asynchronous method VLASH by 14.8% in average success rate. It maintains high accuracy even as inference latency ( $\Delta$ ) increases, whereas VLASH and RTC (Reinforced Trajectory Correction) degrade significantly.
Real-World Performance: In real-world experiments involving complex tasks like folding clothes, Jetson-PI on a Jetson Orin achieved performance comparable to an RTX 4090 setup, with negligible accuracy loss compared to the high-end baseline.
Latency Breakdown: System optimizations (graph reuse, buffering, unrolling) reduced total reaction time from ~1.4s to ~0.16s on Jetson Orin.

Significance and Claims

The paper claims that Jetson-PI provides a practical solution for deploying large-scale VLA models on low-power, battery-operated mobile robots. By addressing the specific bottlenecks of edge computing—namely bandwidth constraints and the temporal gap between perception and execution—the method enables high-frequency, responsive robot control without the power penalties of high-end GPU clusters. The authors emphasize that while onboard platforms are fundamentally limited compared to GPU clusters, Jetson-PI offers a viable path to expand VLA applications from laboratory demos to real-world, mobile scenarios.