Learning to Race in Minutes: Infoprop Dyna on the Mini Wheelbot

Technical Summary: Learning to Race in Minutes with Infoprop Dyna on the Mini Wheelbot

Problem Statement
Controlling robotic systems characterized by fast, unstable, and nonlinear dynamics remains a significant challenge, particularly when learning directly on real hardware. While Reinforcement Learning (RL) offers potential, conventional methods often suffer from poor sample efficiency, necessitating the use of carefully designed physics-based simulators and extensive domain randomization to achieve successful sim-to-real transfer. Model-based RL (MBRL) improves sample efficiency by learning a dynamics model to augment real experience with imagined rollouts; however, imperfect models often lead to "model exploitation" during long-horizon planning, degrading real-world performance. The specific challenge addressed in this work is the Mini Wheelbot, an underactuated unicycle robot with highly nonlinear yaw dynamics. Its control is complicated by ground contact interactions, small time constants, and sensitivity to actuation, making accurate modeling difficult and long-horizon planning essential for effective trajectory tracking.

Methodology
The authors employ Infoprop Dyna, a state-of-the-art, uncertainty-aware MBRL framework, to enable the Mini Wheelbot to learn directly from real-world interactions without relying on a physics-based simulator.

• Core Algorithm: Infoprop Dyna treats model predictions as noisy observations of a ground truth signal. By leveraging information theory, the framework recovers the estimated ground truth signal while simultaneously tracking accumulated corruption due to noise. This approach mitigates model exploitation and enables reliable rollouts over longer horizons than prior methods.
• Dynamics Modeling: Unlike previous benchmarks that assume fully observed, memoryless states, the Mini Wheelbot relies on onboard state estimation. Consequently, the learned transition model is conditioned on a short history of estimated physics states and applied control inputs to predict the next estimated state. This allows the model to implicitly capture the dynamics of the state estimator alongside the physical system.
• Training Pipeline: The system utilizes a distributed training setup (illustrated in Figure 2). Real-world data is collected by the physical robot and transmitted via SSH to a local workstation, which forwards the dataset to a High-Performance Computing (HPC) cluster. Model learning and policy optimization are performed using a JAX-based implementation on top of BRAX. Crucially, no simulation model is used for training; the HPC cluster generates large-scale parallel synthetic rollouts using the learned dynamics model, replacing the need for a physics simulator.
• Initialization: To ensure safety and provide informative transitions, the real-world dataset is warm-started using one minute of data collected by manually driving the robot with an approximate model predictive controller (AMPC) and a joystick interface.

Key Contributions

1. Simulator-Free Real-World Learning: The work demonstrates that Infoprop Dyna can enable a robot to learn a complex control policy directly from real-world interactions, bypassing the need for carefully engineered sim-to-real pipelines.
2. Rapid Acquisition of Aggressive Behavior: The Mini Wheelbot learns to race around a fixed track within 11 minutes of real-world experience, progressing from conservative motion to aggressive racing strategies.
3. Discovery of Non-Trivial Strategies: The learned policy discovers complex behaviors, such as controlled slipping at high speeds to navigate corners, which are difficult to capture using purely physics-based simulations.
4. Distributed Training Architecture: The paper presents a pipeline where training times are comparable to real-world data collection times through massive parallelization on HPC, contrasting with prior benchmarks where training dominated wall-clock time.

Results
In a single continuous training run, the agent successfully completed its first lap after six minutes of real-world interaction. By the 11-minute mark, the agent achieved peak performance, completing multiple laps with high consistency.

• Performance Metrics: The final racing agent achieved an average speed of 0.5 m/s and a peak speed of 0.97 m/s. In comparison, the initial AMPC baseline had an average speed of 0.15 m/s and a maximum speed of 0.33 m/s.
• Efficiency: The racing agent completed more than three times the number of laps as the AMPC agent.
• Behavioral Differences: While the AMPC relied on gyroscopic effects to reorient, the learned agent exploited controlled slipping to handle fast corners, allowing it to stick more closely to the track centerline and maintain higher velocities.

Significance
The paper claims that these results highlight the practical potential of Infoprop Dyna to bridge the gap between data efficiency and high final performance on real robotic systems. By demonstrating that a challenging, underactuated system can learn aggressive racing behaviors in minutes without a simulator, the work suggests a viable path toward wider adoption of real-world model-based reinforcement learning. The authors position this as a step toward scaling such methods to larger arenas and arbitrary track layouts, as well as developing modular interfaces for broader application across robotic systems.