💻 computer science

RLinf-USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI

This paper presents USER, a unified and extensible system that treats physical robots as first-class resources to overcome the unique challenges of real-world online policy learning through a novel hardware abstraction layer, adaptive communication plane, and fully asynchronous learning framework.

Original authors: Hongzhi Zang, Shu'ang Yu, Hao Lin, Tianxing Zhou, Zefang Huang, Zhen Guo, Xin Xu, Jiakai Zhou, Yuze Sheng, Shizhe Zhang, Feng Gao, Wenhao Tang, Yufeng Yue, Quanlu Zhang, Xinlei Chen, Chao Yu, Yu Wang

Published 2026-02-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongzhi Zang, Shu'ang Yu, Hao Lin, Tianxing Zhou, Zefang Huang, Zhen Guo, Xin Xu, Jiakai Zhou, Yuze Sheng, Shizhe Zhang, Feng Gao, Wenhao Tang, Yufeng Yue, Quanlu Zhang, Xinlei Chen, Chao Yu, Yu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to do chores, like folding laundry or making coffee. In the past, scientists usually taught robots in a video game simulation. It's like practicing piano in a silent room where you can hit the wrong note, instantly rewind time, and try again a million times per second.

But the real world is different. You can't rewind a spilled cup of coffee. You can't speed up time. And if you have 10 different types of robots (some with wheels, some with arms, some with cameras), they all speak different "languages" and live in different places (some in the cloud, some in a factory).

This paper introduces USER, a new "operating system" designed specifically to teach robots in the real world efficiently. Think of USER as a universal translator and traffic controller that makes real-world robot learning possible without the chaos.

Here is how USER works, broken down with simple analogies:

1. The "Universal Remote" for Robots (Unified Hardware Abstraction)

The Problem: Usually, a computer program sees a GPU (a super-fast brain chip) as a brain and a robot arm as a "toy" or an "external object." It's hard to tell them to work together.
The USER Solution: USER treats the robot arm exactly the same way it treats the GPU.

Analogy: Imagine a conductor of an orchestra. Before, the conductor only knew how to talk to the violinists (the GPUs). The drummers (the robots) were standing in the back, shouting instructions that no one understood.
With USER: The conductor now has a "Universal Remote." Whether it's a GPU, a robot arm, or a camera, USER sees them all as "musicians" on the same stage. It can automatically find them, assign them tasks, and tell them exactly when to play, regardless of what brand or model they are.

2. The "Smart Tunnel" for Data (Adaptive Communication)

The Problem: Real-world robots often live in different places. Some are in a factory (Edge), and the super-computer training them is in a data center (Cloud). Sending data back and forth across the internet is slow, like trying to pour a bucket of water through a straw.
The USER Solution: USER builds a "Smart Tunnel" that optimizes how data travels.

Analogy: Imagine a busy highway. If every car (data packet) has to drive all the way to the city center (Cloud) and back out to the suburbs (Edge) just to drop off a letter, traffic jams happen.
With USER: It creates a "local express lane." If two robots are in the same factory, they talk directly to each other without going to the cloud. If they must talk to the cloud, USER uses a "tunnel" (like a private, encrypted bridge) to make sure the data doesn't get stuck in traffic. It also limits how much data the "brain" (GPU) has to process at once so it doesn't get overwhelmed.

3. The "Asynchronous Assembly Line" (Fully Asynchronous Pipeline)

The Problem: In old systems, the robot had to stop moving while the computer thought about what to do next. It's like a chef stopping to chop vegetables, then waiting for the oven to preheat, then waiting for the dishwasher to finish, all while the food sits on the counter getting cold.
The USER Solution: USER makes everything happen at the same time.

Analogy: Think of a high-speed sushi restaurant.
- Old Way: The chef cuts the fish, waits for the rice to cook, waits for the plate to be washed, then serves.
- USER Way: One person is cutting fish, another is cooking rice, another is washing plates, and a fourth is serving, all at the exact same time. The robot never stops moving; it keeps gathering data while the computer learns in the background. If the computer crashes or the internet flickers, the robot keeps working, and the system picks up right where it left off.

4. The "Infinite Backpack" (Persistent-Cache-Aware Buffer)

The Problem: Robots generate massive amounts of video data. Computers usually try to keep this data in their "short-term memory" (RAM). But RAM is small and expensive. If you run out of space, you have to delete old data, which is like throwing away a lesson you learned yesterday.
The USER Solution: USER uses a "Smart Backpack" that mixes short-term and long-term storage.

Analogy: Imagine you are studying for a huge exam.
- Old Way: You only keep the last 5 flashcards in your hand. If you need to review a card from yesterday, you have to go back to the library to find it, which takes forever.
- With USER: You have a "Magic Backpack." The most important cards (recent data) are in your hand for quick access. But the rest of the cards are neatly organized in the backpack (on a hard drive). If you need an old card, the system instantly pulls it out without you having to leave your seat. This allows the robot to learn from weeks of experience, not just the last few minutes.

5. The "Swiss Army Knife" for Learning (Extensible Framework)

The Problem: Different robots need different learning styles. Some need to learn by trial and error (Reinforcement Learning), others by watching humans (Imitation Learning), and some need to understand complex language and vision (VLA models).
The USER Solution: USER is built like a modular Lego set.

Analogy: Instead of building a new car engine for every new car model, USER is a universal engine block. You can plug in a "CNN" (a simple visual brain), a "Generative Flow" (a creative brain), or a "Large VLA" (a super-intelligent brain) without rebuilding the whole car. It supports rewards from rules, humans, or AI, making it flexible for any task.

Why Does This Matter?

Before USER, teaching robots in the real world was slow, fragile, and limited to simple tasks.

Without USER: "Let's try to teach this robot to pick up a cup. Oh no, the network lagged, the robot stopped, and we lost all our data. Let's start over."
With USER: "The robot is picking up cups, the system is learning in the background, the data is being saved safely, and even if the internet flickers, the robot keeps working. We can train 10 different robots at once, and they learn faster."

In short: USER turns the chaotic, slow, and fragile process of real-world robot learning into a smooth, efficient, and robust operation, paving the way for robots that can actually learn and adapt in our messy, real world.

1. Problem Statement

While simulation-based training is standard for embodied AI, the "Sim-to-Real" gap often leads to policy degradation. Consequently, online policy learning directly in the physical world is increasingly necessary. However, real-world learning faces unique system-level challenges that make it fundamentally different from simulation:

Non-acceleratable Time: Physical robots operate in real-time; experiments cannot be sped up, arbitrarily reset, or massively replicated.
Heterogeneity & Distribution: Deployments often involve diverse robot types (different kinematics, sensors) and distributed infrastructure (Cloud-Edge), leading to network instability and bandwidth asymmetry.
Data Efficiency & Long Horizons: Physical interaction is slow, making data collection the bottleneck. Long-horizon tasks generate massive amounts of high-dimensional visual data, which existing memory-centric buffers cannot handle efficiently or persistently.
System Coupling: Current approaches often treat robots as external environments rather than integrated hardware resources, leading to inefficient scheduling and synchronization issues between data generation and training.

2. Methodology: The USER System

The authors propose USER (Unified and extensible SystEm for Real-world online policy learning), a system designed to treat physical robots as first-class hardware resources alongside GPUs. The architecture consists of three core pillars:

A. Unified Hardware Abstraction Layer (HAL)

Concept: Virtualizes physical robots, GPUs, and peripherals (cameras, sensors) as schedulable "hardware units" within a single cluster.
Functionality:
- Discovery & Registration: Uses pluggable checkers to automatically discover heterogeneous hardware (e.g., different robot models, GPUs) and attach metadata (network identity, sensor bindings).
- Scheduling: Maps process ranks to specific hardware units (e.g., binding rollout workers to specific robots and training workers to specific GPUs) regardless of the underlying heterogeneity.
- Node Groups: Organizes nodes into groups based on hardware homogeneity to enable flexible scheduling policies.

B. Adaptive Communication Plane

Designed to handle the complexities of Cloud-Edge deployments across isolated network domains (e.g., NATs, VLANs).

Tunneling-based Networking: Uses UDP tunneling to create a flattened TCP/IP substrate, allowing bidirectional communication between isolated Cloud and Edge nodes without direct routing issues.
Distributed Data Channels: Replaces centralized data routing with a FIFO producer-consumer queue system. Channels are sharded by data keys (e.g., Robot ID) to localize traffic within Edge or Cloud regions, minimizing cross-domain bandwidth usage and latency.
SM-Aware Weight Synchronization: Addresses the contention between training and rollout. Since NCCL (NVIDIA Collective Communications Library) consumes Streaming Multiprocessors (SMs), USER implements a tunable cap on NCCL CTAs to prevent weight synchronization from monopolizing GPU resources and degrading rollout latency.

C. Fully Asynchronous Learning Framework

Decouples data generation, training, and synchronization to maximize efficiency.

Asynchronous Pipeline: Rollout workers execute policies on robots continuously without waiting for training updates. Training workers sample data asynchronously.
Persistent-Cache-Aware Buffer:
- Design: A hybrid storage system where recent data resides in a bounded in-memory cache (FIFO replacement) for high-throughput sampling, while historical data is asynchronously persisted to disk.
- Benefits: Supports arbitrarily large datasets, enables robust crash recovery, and allows for the reuse of historical data across different policy versions (off-policy learning).
Extensible Abstractions: Provides unified interfaces for:
- Policies: From lightweight CNN/MLP controllers to generative flow-matching policies and large Vision-Language-Action (VLA) models (e.g., $\pi_0$ ).
- Algorithms: Supports both Reinforcement Learning (SAC, RLPD) and Imitation Learning (HG-DAgger).
- Rewards: Supports rule-based, human-provided, and learned reward models.

3. Key Contributions

Unified Hardware Abstraction: Treats robots as first-class resources alongside accelerators, enabling automatic discovery and scheduling of heterogeneous robot fleets.
Adaptive Communication Plane: Introduces tunneling-based networking, distributed data channels, and SM-aware synchronization to solve Cloud-Edge bandwidth and latency issues.
Asynchronous Framework with Persistent Buffer: Moves beyond memory-centric pipelines to a persistent, cache-aware design that supports long-horizon, crash-resilient learning.
Unified Execution Pipeline: Enables diverse learning paradigms (RL, IL, Human-in-the-loop) and model scales (small CNNs to large VLAs) within a single system.
Open-Source Implementation: Provides a reusable systems foundation for the community.

4. Experimental Results

The system was evaluated on five real-world manipulation tasks (Peg Insertion, Charger Plugging, Cap Tightening, Pick-and-Place, Table Clean-up) using Franka and ARX robotic arms.

Performance Gains:
- Asynchronous Efficiency: The asynchronous pipeline improved generation throughput by 1.20x–1.55x and training throughput by 4.61x–5.70x compared to synchronous baselines.
- Convergence Speed: On the Peg Insertion task, the asynchronous pipeline reduced convergence time from 8,000+ seconds to ~1,500 seconds.
- VLA Fine-tuning: For the $\pi_0$ VLA model, online training (HG-DAgger) improved success rates from 39/60 to 58/60 (Pick-and-Place) and 9/20 to 16/20 (Table Clean-up) in short durations.
Scalability & Heterogeneity:
- Successfully trained a unified policy on two heterogeneous robots (7-DoF Franka and 6-DoF ARX) simultaneously, demonstrating cross-embodiment learning.
- Multi-robot parallel training achieved convergence speeds matching single-robot baselines.
Communication Efficiency:
- In cross-domain (Cloud-Edge) settings, the distributed data channel reduced total episode generation time by ~3x compared to centralized routing by localizing traffic.
Buffer Performance: The persistent-cache-aware buffer achieved throughput significantly higher than pure disk buffers while supporting datasets far larger than RAM capacity.

5. Significance

USER represents a paradigm shift in embodied AI research by addressing the systems bottleneck rather than just the algorithmic one.

Bridging the Gap: It provides the necessary infrastructure to make real-world online learning feasible, reliable, and scalable, moving beyond the limitations of simulation.
Enabling Large Models: By solving communication and data management challenges, USER makes it possible to train and deploy large-scale VLA models directly on physical robots in the real world.
Community Impact: The open-source nature of USER lowers the barrier to entry for real-world robotics research, allowing researchers to focus on algorithmic innovation rather than infrastructure engineering.

In summary, USER establishes a unified, extensible, and robust foundation for the next generation of embodied intelligence, where physical robots are integral components of the distributed learning loop.