🤖 AI

rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference

rMuscle is a real-time Vision-Language-Action inference framework inspired by human muscle memory that accelerates robot responsiveness by 1.29–1.42× through a dual-phase caching mechanism that reuses visual tokens and neuron activations across similar repeated tasks while maintaining original success rates.

Original authors: Kaijun Zhou, Zhiyang Li, Le Chen, Jinyu Gu

Published 2026-09-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaijun Zhou, Zhiyang Li, Le Chen, Jinyu Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the bustling world of modern manufacturing, a new kind of worker is taking its place on the factory floor: the robot guided by artificial intelligence. Unlike the rigid, pre-programmed machines of the past that could only repeat a single motion forever, these new systems are designed to understand the world through sight and language. They can look at a cluttered table, read an instruction like "pick up the red bottle," and figure out the precise movements needed to complete the task. This capability relies on a specific type of brain for the machine, known as a vision-language-action model. These models act as the central nervous system, processing what the robot sees and hears, then translating that understanding into a stream of physical commands. For these robots to be truly useful in a real factory, they must be fast. If the robot's brain takes too long to think, the machine stutters, its movements become jerky, and it fails to react quickly enough when a human worker steps in or an object shifts. The speed at which the robot can think and decide is the single biggest factor determining whether it can keep up with the pace of human work.

Researchers at Shanghai Jiao Tong University have identified a fundamental bottleneck in how these intelligent robots currently think. They found that while the software driving these robots is incredibly sophisticated, it is often wasting time by relearning the same things over and over again. In a typical factory setting, a robot does not face a completely new world every second. Instead, it performs a repetitive loop of tasks, often moving similar objects to similar spots under similar lighting conditions. The team discovered that because the tasks are so similar, the robot's internal state—the complex patterns of activity inside its digital brain—also becomes remarkably similar from one attempt to the next. Just as a human pianist does not need to relearn the finger movements for a familiar song every time they play it, the robot is capable of recalling these patterns. However, existing software frameworks do not take advantage of this; they force the robot to perform every single calculation from scratch, even when the answer is essentially the same as it was a moment ago.

To solve this, the researchers developed a new system called rMuscle, a framework designed to give robots a form of digital muscle memory. The system works by watching how the robot performs a task and saving the "thoughts" it had during successful attempts. When the robot encounters a situation that looks like a previous one, rMuscle does not start from zero. Instead, it reaches into its memory bank and retrieves the relevant internal patterns. The system is built on two distinct types of memory to handle the different ways the robot thinks. The first part, called the Context Cache, handles the visual processing. When the robot sees a familiar scene, this cache allows it to skip the heavy lifting of analyzing every single pixel again, reusing the results of previous visual analyses instead. The second part, the Action Cache, handles the decision-making for movement. It recognizes that in many repetitive tasks, only a small, specific set of the robot's internal decision-makers are actually needed to solve the problem. By identifying which specific parts of the brain are active in a familiar scenario, the system can ignore the rest, loading only the necessary components to make a decision.

The researchers tested this approach on a variety of robots and tasks, ranging from simulated environments to real-world physical robots working on assembly lines. They used powerful computers and specialized hardware often found in high-end robotics labs to see how much faster the robots could think. The results showed a significant improvement in speed. On standard high-performance computers, the new system made the robots think about 29% faster. On specialized, smaller computers designed to fit inside robots themselves, the speed increase was even more dramatic, reaching up to 42% faster. This means a robot that previously took a fraction of a second to decide its next move could now do it almost instantly, allowing for smoother, more fluid motion. Crucially, this speed did not come at the cost of accuracy. The robots did not start making mistakes or dropping objects; they maintained the same high success rate as before, simply reaching their decisions more efficiently.

The success of rMuscle relies on a clever way of managing memory to ensure the robot does not get bogged down by the very data it is trying to reuse. Storing every single thought the robot ever had would require too much space, so the system is designed to be selective. It keeps a small, sliding window of the most relevant recent memories, discarding older ones that are unlikely to be needed again. When the robot needs to recall a memory that is not currently in its active window, the system reconstructs it on the fly while the robot is physically moving its arms. This means the robot is always working, using the time it spends moving to prepare the next set of thoughts for when it stops to think again. This seamless integration ensures that the robot never has to wait for its memory to catch up.

The implications of this work extend beyond just making robots faster; it changes how we can deploy them in the real world. By reducing the time it takes for a robot to process its environment, the system allows for more responsive interactions. A robot can react to a human stepping into its path or a part falling off a conveyor belt much more quickly, making it safer and more reliable to work alongside people. The researchers demonstrated this with physical robots performing complex assembly tasks, such as packing furniture components onto a moving belt or handling delicate bottles with two arms working in unison. In every case, the system preserved the robot's ability to succeed while shaving off critical milliseconds from each decision cycle.

This approach represents a shift in how we build intelligent machines. Rather than trying to make the underlying brain of the robot more powerful or complex, the researchers focused on how that brain is used. They realized that the intelligence required for factory work is not about solving a new puzzle every second, but about efficiently reusing solutions to a familiar set of problems. By building a system that respects the repetitive nature of industrial work and leverages the similarities between tasks, they have created a way for robots to move with a fluidity that was previously out of reach. The work suggests that the future of efficient robotics lies not just in better hardware, but in smarter ways of managing the flow of information, allowing machines to remember their past successes and apply them to the present moment with minimal effort.

Technical Summary: rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference

1. Problem Statement

Vision-Language-Action (VLA) models have emerged as the dominant policy paradigm for embodied AI, particularly in factory settings where robots perform repetitive manual tasks. However, the inference latency of these models directly limits robot responsiveness and motion smoothness. Existing inference frameworks (e.g., FlashRT, vla.cpp) and acceleration techniques (e.g., kernel fusion, quantization, cross-frame token reuse) fail to fully exploit the unique characteristics of embodied workloads.

The authors identify two primary limitations in current approaches:

Failure to Exploit Cross-Execution Similarity: Industrial robots operate at fixed workstations performing recurring tasks. While existing methods focus on similarity within a single frame or a single denoising phase, they ignore the substantial similarity in observations, action trajectories, and internal model states across repeated executions of the same task.
Ignoring Heterogeneous Bottlenecks: VLA inference consists of two distinct stages with different performance bottlenecks. In models like $\pi_{0.5}$ , the Vision-Language Model (VLM) prefill phase is compute-bound (dominated by MLP computations), while the action denoising phase is memory-bound (dominated by weight loading for short sequences). Optimizing only one stage yields limited end-to-end speedup.

2. Methodology: rMuscle

Inspired by human muscle memory—where similar contexts trigger the retrieval of relevant prior experiences—rMuscle is a real-time VLA inference framework designed to exploit cross-execution similarity through a dual-phase cache.

2.1 Dual-Phase Cache Design

The framework targets the two distinct bottlenecks of VLA inference:

Context Cache (Compute-Bound Optimization):
- Target: Accelerates the VLM prefill phase.
- Mechanism: Exploits similarity in inputs to latency-critical MLP layers. Instead of fully recomputing similar visual tokens, rMuscle reuses cached FFN outputs from a reference execution.
- Selective Recomputation: It calculates an "input drift" metric to identify visual tokens that have changed substantially. Only these tokens are recomputed; the rest are retrieved from the cache. Instruction and robot-state rows are always recomputed.
- Result: Reduces computation while preserving model accuracy.
Action Cache (Memory-Bound Optimization):
- Target: Accelerates the action denoising phase.
- Mechanism: Exploits the observation that similar tasks activate overlapping sets of "important" neurons.
- Neuron Sparsity: Instead of loading all weights, rMuscle uses a reference execution's activation pattern to predict which neurons are important. It loads and computes only the weights for these selected neurons.
- Approximation: The remaining neurons are approximated using outputs from an earlier fully computed "anchor" step.
- Group-Shared Masks: To reduce weight gathering overhead, consecutive denoising steps are grouped, sharing a single important-neuron mask.

2.2 Cache Management

To ensure low overhead and manageable memory footprint, rMuscle employs three key techniques:

Asynchronous Online Reconstruction: Expanded VLM states (FFN inputs/outputs) are too large to store for all references. rMuscle stores only compact inputs and reconstructs the full states during physical robot execution (overlapped with action execution), ensuring the reconstruction does not block the inference critical path.
Sliding-Window Prefetching: Leveraging the temporal locality of robotic workflows, the system maintains a sliding window of relevant cache entries in GPU memory. As the robot executes, the window updates dynamically to prefetch likely future references.
Mask Sharing: Consecutive denoising steps within a group share an important-neuron mask, minimizing the cache memory footprint and access overhead.

3. Key Contributions

Workload Characterization: The authors demonstrate that repeated task executions exhibit high similarity not only in observations and trajectories but also in internal model states (FFN inputs/outputs and neuron activation patterns).
Dual-Phase Inference System: They design an end-to-end system with a Context Cache for compute-bound VLM prefill and an Action Cache for memory-bound denoising, addressing the specific bottlenecks of SOTA VLA models.
Comprehensive Evaluation: The system is evaluated on three models ( $\pi_{0.5}$ , GR00T N1.6, X-VLA) across 92 embodied tasks (LIBERO, RoboTwin, and physical robots) on both RTX 4090 and Jetson Thor.

4. Results

Speedup: rMuscle achieves 1.29–1.42× end-to-end speedup over the state-of-the-art (FlashRT) inference engine.
- On RTX 4090: Up to 1.29× for $\pi_{0.5}$ , 1.20× for GR00T N1.6, and 1.50× for X-VLA.
- On Jetson Thor: Up to 1.42× for $\pi_{0.5}$ , 1.23× for GR00T N1.6, and 1.43× for X-VLA.
Policy Quality: The method maintains the original success rates (SR) across simulation and real-world tasks.
- On RoboTwin (50 tasks), rMuscle matches the vanilla success rate of 35.2%, outperforming DP-Cache (33.9%) and NIRVANA (26.9%).
- On physical tasks (ALOHA dual-arm pick-and-place, DOBOT/Franka assembly-line packing), rMuscle matches vanilla performance (76% and 84% SR, respectively), significantly outperforming baselines that suffer from reduced accuracy due to aggressive step-skipping.
Memory Overhead: The framework adds minimal overhead. On RTX 4090, GPU memory usage increases by less than 13% (from 8.4 GB to ~9.4–11.4 GB). On Jetson Thor, unified memory occupancy increases by at most 6.0 percentage points.

5. Significance and Claims

The paper claims that rMuscle represents the fastest VLA inference engine optimized specifically for embodied workloads. Its significance lies in:

Exploiting Task Repetition: Moving beyond single-frame or single-step optimizations to leverage the inherent redundancy in repeated industrial tasks.
Holistic Optimization: Addressing both compute-bound and memory-bound stages of VLA inference simultaneously, rather than optimizing them in isolation.
Practical Viability: Demonstrating that significant latency reductions can be achieved without compromising the success rates required for real-world deployment on physical robots.

The authors conclude that by reducing per-call latency, rMuscle improves robot responsiveness, motion smoothness, and task throughput, enabling fixed-workstation robots to complete more tasks per hour on existing hardware.