🤖 machine learning

RLDX-1 Technical Report

This paper introduces RLDX-1, a general robotic policy based on the Multi-Stream Action Transformer (MSAT) architecture that integrates heterogeneous modalities and specialized system designs to significantly outperform existing Vision-Language-Action models in complex, contact-rich, and dynamic real-world dexterous manipulation tasks.

Original authors: Dongyoung Kim, Huiwon Jang, Myungkyu Koo, Suhyeok Jang, Taeyoung Kim, Beomjun Kim, Byungjun Yoon, Changsung Jang, Daewon Choi, Dongsu Han, Donguk Lee, Heeseung Kwon, Hojin Jeon, Jaehyun Kang, Jaekyoun

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongyoung Kim, Huiwon Jang, Myungkyu Koo, Suhyeok Jang, Taeyoung Kim, Beomjun Kim, Byungjun Yoon, Changsung Jang, Daewon Choi, Dongsu Han, Donguk Lee, Heeseung Kwon, Hojin Jeon, Jaehyun Kang, Jaekyoung Bae, Jihyuk Lee, Jimin Lee, John Won, Joonwoo Ahn, Junhyeong Park, Junyoung Sung, Kyungmin Lee, Minseong Han, Minsung Yoon, Sejune Joo, Seonil Son, Seungcheol Park, Seunggeun Cho, Seungjun Moon, Seungku Kim, Yonghoon Dong, Yongjin Cho, Youngchan Kim, Chang Hwan Kim, Dohyeon Kim, Hazel Lee, Heecheol Kim, Hensen Ahn, Hyungkyu Ryu, Hyunsoo Choi, Hyunsoo Shin, Jaeheon Jung, Jaewoo Kim, Jinwook Kim, Joochul Chang, Joonsoo Kim, Junghun Park, Jungwoo Park, Junho Cho, Junhyeok Park, Junwon Lee, Kangwook Lee, Kwanghoon Kim, Kyoungwhan Choe, Manoj Bhadu, Nayoung Oh, Sangjun Kim, Sangwoo Kim, Seunghoon Shim, Seunghyun Kim, Seungjun Lee, Seungyup Ka, Sungryol Yang, Wook Jung, Yashu Shukla, Yeonjae Lee, Yeonwoo Bae, Jinwoo Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a robot that is incredibly smart at reading instructions and looking at images. It knows what a "cup" is, what "pouring" means, and can speak fluently about them. This is like a robot with a PhD in theory, but without experience in the real world. If you ask it to catch a ball moving on a conveyor belt, or to twist a lightbulb without breaking it, it might freeze because it does not understand movement, memory, or touch.

The work introduces RLDX-1, a new robot "brain" designed to fix this problem. Consider RLDX-1 not just as an intelligent reader, but as a master craftsman who combines three new superpowers with its existing intelligence:

1. The Superpowers: Movement, Memory, and Touch

Motion Perception (The "Juggler"):
Most robots look at a still photo and guess what to do next. But the real world moves. If a box slides on a conveyor belt, a static photo is useless. RLDX-1 is like a juggler who does not just look at the ball; it watches the entire video of the ball flying through the air. It understands speed and direction, enabling it to catch moving objects that other robots miss.
Long-Term Memory (The "Sherlock Holmes"):
Some tasks take time. Imagine a shell game where a person hides a die under one of three cups, walks away, and then asks the robot to find it. A robot with only "short-term memory" sees the cups but forgets which one the person touched. RLDX-1 keeps a "notebook" about what happened seconds or minutes ago. It remembers the sequence of events, allowing it to solve puzzles that require looking back into the past.
Physical Sensing (The "Gentle Hand"):
Seeing is great, but you cannot see inside a socket or feel how hard you are pressing a fragile egg. RLDX-1 can "feel" through its sensors. It reads the torque (rotational force) in its joints and tactile (touch) signals from its skin. This is like a blind person who can say exactly when a plug is seated in a socket or when an egg is about to break, and immediately adjust their grip.

2. The Training: How it Learned

You cannot simply teach a robot these skills by showing it a few videos. RLDX-1 went through a three-stage training camp:

Stage 1: The Generalist (Pre-training): It watched millions of videos of various robots (arms, hands, humanoids) doing simple things. This gave it a broad understanding of how the world works.
Stage 2: The Specialist (Mid-training): Here, it learned its new superpowers. It practiced specifically with the modules for "movement," "memory," and "touch." To do this, researchers used a clever trick: they employed AI video generators to create thousands of new, rare scenarios (like a robot pouring soup in a kitchen that does not exist), and then used mathematics to figure out what the robot should have done. This filled the gaps where real data was missing.
Stage 3: The Refinement (Post-training): Finally, it practiced on specific real-world tasks and learned from its own mistakes. If it failed to pick up a cup, it tried again, using a reinforcement learning method that works like a coach giving immediate feedback to improve performance.

3. The Speed: On the Fast Track

Even the smartest brain is useless if it is too slow. If a robot takes 0.1 seconds to think, the world may have changed by the time it acts. The work explains that RLDX-1 was optimized to run 1.6 times faster than standard systems.

The Analogy: Imagine a delivery driver who stops at every single traffic light to check a map (standard processing). RLDX-1 is like a driver who has mapped the entire route in advance, knows exactly which lights will turn green, and drives through without stopping. This allows it to react in real time, even on fast-moving conveyor belts.

4. The Results: The Proof is in the Pudding

The researchers tested RLDX-1 in two ways against other top robot brains (such as $\pi0.5$ and GR00T N1.6):

In Simulation (The Video Game): RLDX-1 consistently beat the others in complex kitchen tasks and challenges involving moving objects.
In the Real World (The Test):
- The Conveyor Belt: When objects moved quickly, RLDX-1 succeeded in 87.5% of cases, while the next-best robot failed in almost 70% of cases.
- The Shell Game: When asked to find a hidden object based on memory, RLDX-1 was correct in 91.7% of cases. The others guessed randomly and were correct only about 30% of the time.
- The Lightbulb: When asked to twist a lightbulb until it lit up, RLDX-1 learned to do it in fewer attempts than a human trainer could demonstrate.

Summary

RLDX-1 is a robot strategy that goes beyond mere "seeing and understanding" toward skilled, craft-like action. By combining a powerful visual brain with specialized modules for movement, memory, and touch, and by training on a massive mix of real and AI-generated data, it achieves human-like abilities in complex, dynamic, and delicate tasks where earlier robots struggled. It is a significant step toward robots that can truly work alongside us in the chaotic, moving, and tactile real world.

Technical Summary: RLDX-1

Problem Statement

While Vision-Language-Action Models (VLAs) have demonstrated significant progress in "versatile intelligence"—the ability to understand diverse visual scenes, follow language instructions, and generalize across tasks—they often struggle with complex manipulation tasks in the real world that require broader functional capabilities. Specifically, existing VLAs lack explicit mechanisms for motion awareness (acting in dynamic environments with moving objects), long-term memory (reasoning beyond short timeframes over extended interaction histories), and physical perception (reasoning about contact forces and torques under occlusion or subtle visual changes). These limitations hinder the deployment of generalist robots in dynamic, contact-rich, and long-horizon scenarios.

Methodology

The authors introduce RLDX-1, a general robotic policy designed for human-like, dexterous manipulation. The system integrates diverse functional capabilities into a unified framework encompassing architecture, data, training, and inference.

1. Neural Architecture: Multi-Stream Action Transformer (MSAT)

RLDX-1 is based on a Flow-Matching VLA architecture but is extended with three key functional modules and a novel transformer design:

Motion Awareness: The Vision-Language Model (VLM) backbone is augmented with a Motion Module that integrates Spatio-Temporal Self-Similarity (STSS) into the Vision Encoder to capture temporal dynamics. Additionally, past video frames are compressed into a single context token to model temporal context efficiently.
Long-Term Memory: An explicit Memory Module manages a queue of past cognitive features, enabling the model to reason over long-term observation histories (e.g., for shell games or multi-stage tasks).
Physical Perception: A dedicated Physics Stream processes physical signals (torque and tactile data) separately from visual and linguistic inputs. This stream predicts future physical signals alongside actions, encouraging the model to internalize contact dynamics.
MSAT Architecture: To handle these heterogeneous modalities, the authors propose the Multi-Stream Action Transformer. It assigns dedicated streams for Cognition (visual/linguistic), Action (proprioception/noisy actions), and Physics (tactile/torque). These streams interact via shared self-attention while maintaining modality-specific parameters, enabling cross-modal information exchange without losing representational fidelity.

2. Training Data and Pipeline

The training strategy relies on a three-stage pipeline and a diverse data mixture:

Data Sources: The model is trained on a combination of (a) large-scale public real-world datasets (Open-X-Embodiment, DROID, Galaxea, Agibot World, etc.), (b) internal data from the ALLEX humanoid and the sensor-enhanced Franka Research 3 (FR3) platforms, and (c) synthetic data.
Synthetic Data Generation: To address data scarcity for rare dexterous scenarios, the authors employ a pipeline based on video generative models (Image-to-Video) to expand scenes and tasks. This is followed by action labeling using an Inverse Dynamics Model (IDM) and a rigorous two-stage filtering process: Video Quality Filtering (using VLMs) and Motion Consistency Filtering (comparing simulator rollouts with generated videos).
Three-Stage Training:
1. Pre-Training: Training on diverse multi-embodiment data to learn general manipulation priors.
2. Mid-Training: Specialized training on embodiment-specific data (ALLEX and FR3) to instill functional capabilities (motion, memory, physics) using internal and synthetic data.
3. Post-Training: Task-specific fine-tuning, optionally enhanced by Reinforcement Learning (RL) using a VLM-derived critic (RECAP framework) to refine performance on challenging tasks.

3. Inference Optimization

To enable real-time control on physical robots, the authors implement a two-stage inference optimization pipeline:

Graph Capture Optimization: Converts the model into a static CUDA graph to eliminate kernel launch overhead and graph fragmentation.
Kernel Optimization: Develops custom fused kernels (e.g., fusing RMSNorm, RoPE, and Attention) to reduce memory traffic and improve execution efficiency for short prefill workloads.
These optimizations reduce latency per step from ~71.2 ms to 43.7 ms on an NVIDIA RTX 5090, achieving a 1.63x speedup.

Key Results

RLDX-1 was evaluated across simulation benchmarks and real-world experiments on the OpenArm, ALLEX, and Franka Research 3 platforms, outperforming state-of-the-art baselines such as $\pi0.5$ and GR00T N1.6.

Simulation Benchmarks: RLDX-1 achieved the highest success rates across all benchmarks, including LIBERO (97.8%), RoboCasa Kitchen (70.6%), and GR-1 Tabletop (58.7%). Notably, it demonstrated superior robustness in the LIBERO-Plus and SIMPLER variants compared to baselines.
Real-World Versatility (OpenArm): For tasks requiring instruction following and generalization to unseen objects/tasks, RLDX-1 significantly outperformed baselines, achieving 54.2% on unseen objects compared to 37.5% for $\pi0.5$ .
Functional Capabilities (ALLEX & FR3):
- Motion Awareness: In conveyor belt tasks with varying speeds, RLDX-1 achieved an average success rate of 86.8%, while baselines collapsed to ~40% or failed on unseen speeds.
- Long-Term Memory: In "Object-in-Box Selection" and "Shell Game" tasks, RLDX-1 achieved success rates of 91.7% each, whereas baselines performed near random chance (~30–50%).
- Physical Perception: In contact-rich tasks like "Plug Insertion" and "Egg Pick-and-Place," RLDX-1 utilized torque and tactile feedback to achieve success rates of 33.3% and 61.1% respectively, outperforming baselines that struggled with contact forces.
Inference Performance: The optimized pipeline achieved a 1.63x speedup, enabling real-time deployment.

Significance and Claims

The work claims that RLDX-1 represents a significant step toward reliable VLAs for complex, contact-rich, and dynamic dexterous manipulation in the real world. By moving beyond "versatile intelligence" and explicitly integrating motion awareness, long-term memory, and physical perception into a unified end-to-end architecture, RLDX-1 addresses critical gaps in current generalist robot policies.

The authors emphasize that their approach is not merely a scaling of existing VLA models but a structural integration of functional capabilities necessary for human manipulation. The success of the synthetic data pipeline in augmenting real-world data and the effectiveness of the three-stage training procedure demonstrate a scalable path toward deploying generalist policies on diverse embodiments. The results on the ALLEX humanoid, where RLDX-1 achieved ~90% success on functional tasks compared to ~40% for frontier models, underscore the model's ability to control high-DoF robots under diverse functional requirements.