CycleRL: Sim-to-Real Deep Reinforcement Learning for Robust Autonomous Bicycle Control

Technical Summary: CycleRL: Sim-to-Real Deep Reinforcement Learning for Robust Autonomous Bicycle Control

Problem Statement
Autonomous control of two-wheeled vehicles presents a significant challenge in robotics due to their inherently unstable, underactuated, and nonlinear dynamics. Traditional control strategies, such as PID, Linear-Quadratic Regulators (LQR), and Model Predictive Control (MPC), often struggle with these dynamics. These model-based approaches are typically sensitive to parameter inaccuracies, computationally intensive, and lack robustness against real-world uncertainties and unmodeled disturbances. While Deep Reinforcement Learning (DRL) offers a potential solution by learning control policies directly from interaction, a critical barrier remains: the "sim-to-real" gap. Policies trained in simulation often fail to transfer to physical hardware, particularly for bicycles controlled solely via steering (without reaction wheels), due to the system's extreme sensitivity to minute physical discrepancies and low-speed instability.

Methodology
The authors propose CycleRL, a model-free deep reinforcement learning framework designed to bridge the simulation-to-reality gap for autonomous bicycle control. The methodology is built upon the following components:

• Simulation Environment: Training is conducted within the high-fidelity NVIDIA Isaac Sim environment, which models complex vehicle-terrain nonlinearities, contact dynamics, and sensor properties.
• Reinforcement Learning Formulation: The control problem is formulated as a Markov Decision Process (MDP). The state space consists of instantaneous observations from IMU and encoder data, while the action space controls the steering angle. The authors employ the Proximal Policy Optimization (PPO) algorithm to optimize the policy.
• Composite Reward Function: A tailored reward function guides the learning process, aggregating five components:
  • Survival Reward: Encourages long-term stability.
  • Velocity Tracking Reward: Penalizes deviation from target forward velocity.
  • Heading Tracking Reward: Penalizes deviation from the desired heading.
  • Action Magnitude Penalty: Discourages aggressive control inputs to improve energy efficiency.
  • Action Rate Penalty: Penalizes rapid changes in control inputs to ensure smooth transitions.
• Domain Randomization Strategy: To ensure robustness and facilitate direct sim-to-real transfer without fine-tuning, the authors implement a hierarchical randomization strategy. This includes randomizing physical dynamics (mass, center of mass, friction, actuator gains), initial states (velocity, lean angle), and task commands (target velocity and heading). Gaussian noise is also injected into observations to simulate sensor imperfections.
• System Architecture: The deployed system utilizes a hierarchical control architecture on an NVIDIA Jetson Orin NX, managing input processing, policy inference, and actuator control at a 50 Hz frequency with minimal latency.

Key Contributions

1. Comprehensive DRL Integration: The paper presents a fully integrated, model-free DRL framework for autonomous bicycle control that eliminates the reliance on accurate analytical dynamics models required by conventional methods.
2. Engineered Reward and Randomization: The authors introduce a carefully designed composite reward function and a multi-layered domain randomization strategy. These elements enable high-performance control in simulation and, crucially, reliable direct transfer to physical hardware.
3. Successful Sim-to-Real Transfer: The framework is validated on a custom-built physical bicycle platform. The work demonstrates that the learned policy can maintain robust balance and execute steering maneuvers under diverse real-world conditions, including varying terrains, payloads, and speeds, without additional hardware tuning.

Experimental Results

• Simulation Performance: In simulation, CycleRL achieved a 99.90% balance success rate, significantly outperforming calibrated PID (76.50%), LQR (93.24%), and Modified Linear Control Law (96.43%) baselines. It demonstrated superior recovery times (1.05 s vs. >1.2 s for baselines) and could maintain stability at speeds as low as 1.05 m/s, a regime where classical baselines failed.
• Real-World Deployment: Upon transfer to the physical platform, the system maintained a 95.00% balance success rate. While heading tracking error increased slightly (from 1.15° in sim to 2.04° in reality, attributed to unmodeled IMU noise), velocity tracking remained highly accurate. The system successfully operated on asphalt, gravel, grass, ramps, and under lateral perturbations.
• Ablation Studies: Experiments confirmed that the composite reward components (particularly tracking rewards) and the full domain randomization strategy are critical. Removing randomization led to a significant degradation in tracking precision when deployed, validating the trade-off between peak theoretical performance in simulation and broad adaptability in reality.

Significance and Claims
The paper claims that CycleRL validates Deep Reinforcement Learning as an effective paradigm for autonomous bicycle control, offering superior adaptability over traditional model-based methods. By successfully transferring a policy trained in a randomized simulation to a physical platform without fine-tuning, the work addresses a major engineering challenge in deploying learning-based controllers for unstable, underactuated systems. The authors position this approach as a promising alternative for urban mobility and last-mile logistics, highlighting that while the method shifts engineering effort toward simulator construction and reward shaping, it effectively bypasses the need for complex system identification and analytical modeling. The success of the zero-shot transfer to a scooter platform further suggests the framework's potential generalizability to other two-wheeled vehicles.