🤖 machine learning

Path Following and Stabilisation of a Bicycle Model using a Reinforcement Learning Approach

This paper presents a Reinforcement Learning approach that successfully enables a virtual Whipple bicycle model to simultaneously follow complex paths and maintain lateral stability across a speed range of 2m/s to 7m/s using only steering angle outputs, validated through curriculum learning and explanatory analysis.

Original authors: Sebastian Weyrer, Peter Manzl, A. L. Schwab, Johannes Gerstmayr

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sebastian Weyrer, Peter Manzl, A. L. Schwab, Johannes Gerstmayr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a toddler how to ride a bicycle. You don't just tell them, "Go straight." You have to teach them how to balance, how to lean into turns, and how to steer without falling over. Now, imagine doing this not with a human child, but with a computer program and a virtual bicycle, using a method called Reinforcement Learning (RL).

This paper is essentially the story of how the researchers taught a digital "agent" (a smart computer program) to ride a virtual bicycle perfectly, balancing it while following a winding path, all without any training wheels or physical help.

Here is the breakdown of their journey, using some everyday analogies:

1. The Virtual Test Track

The researchers didn't use a real bike in a garage. They built a virtual bicycle inside a computer. This isn't just a cartoon; it's a highly accurate physics model (called the "Whipple model") that behaves exactly like a real bike.

The Challenge: Real bikes are tricky. If you go too slow, they fall over. If you go too fast, they get wobbly. They also have a weird quirk: to turn left, you actually have to steer slightly right first (this is called "counter-steering").
The Goal: The computer agent had to do two things at once: stay upright (stabilization) and follow a specific line on the ground (path following). It could only control the handlebars. It couldn't push the bike or change the speed; the speed was set by an invisible "cyclist."

2. The Teacher: Reinforcement Learning

Think of the agent as a student and the computer simulation as a very strict teacher.

The Game: The agent tries to ride.
The Scorecard (Reward): Every time the bike stays close to the line and doesn't fall over, the agent gets points (a "reward"). If it falls or goes too far off-track, it gets zero points, and the game restarts.
The Learning: The agent tries millions of times. At first, it crashes constantly. But slowly, it starts to realize, "Oh, when I lean left, I need to steer right to stay up," or "When the path curves, I need to look ahead." It learns by trial and error, not by being told the rules of physics.

3. The Secret Sauce: "Curriculum Learning"

If you tried to teach a toddler to ride a bike on a steep, winding mountain road immediately, they would fail. You start them on a flat, straight driveway.
The researchers used a strategy called Curriculum Learning. This is like a video game that gets harder as you level up:

Level 1: The agent starts on a straight path at a moderate, easy speed where the bike naturally wants to stay upright.
Level 2: As the agent gets good, the teacher introduces curves and changes the speed.
Level 3: The agent faces the hardest challenges: very slow speeds (where the bike is very unstable) and sharp, tight turns.
By starting easy and gradually increasing the difficulty, the agent learned to handle the full range of speeds (from a slow 2 m/s to a fast 7 m/s) and complex paths like slaloms and full circles.

4. The Results: A Master Rider

After millions of virtual rides (taking about 55 hours of simulated time), the agent became an expert.

The Test: They put the trained agent on a "Benchmark Path"—a complex track with circles, zig-zags, and lane changes.
The Outcome: The agent kept the bike upright and followed the path with incredible precision. The average distance between the bike and the line it was supposed to follow was less than 9 centimeters (about the width of a hand).
The Speed: It worked perfectly whether the bike was moving slowly or quickly.

5. Did the Agent Actually "Understand" Physics?

The researchers didn't just want a black box that worked; they wanted to know how it worked. They used a tool called SHAP (which acts like an X-ray for AI decisions) to see what the agent was paying attention to.

What they found: The agent had learned the exact same physics tricks humans use!
- Counter-Steering: The agent learned to steer slightly the opposite way to initiate a turn.
- Leaning into the Fall: When the bike started to tip, the agent learned to steer into the direction of the fall to catch it, just like a real rider does.
- Looking Ahead: The agent paid the most attention to the path a few meters ahead, not just where the wheels were touching the ground.

The Bottom Line

This paper proves that a computer program can learn to ride a bicycle from scratch, balancing it and steering it through complex courses, without any pre-programmed physics equations or human help. It learned the "feel" of the bike through trial and error, eventually becoming a master rider that handles speed changes and tricky turns better than many traditional computer controllers.

The researchers conclude that this approach works, but they note that for a real-world robot bike, they would need to refine the model to handle bumpy roads and figure out how to "see" the path (like using a camera) instead of having the path pre-loaded in the computer.

Technical Summary: Path Following and Stabilisation of a Bicycle Model using a Reinforcement Learning Approach

Problem Statement
Controlling the motion of a bicycle is a complex task due to its speed-dependent lateral stability and non-minimal phase behavior, which requires steering into the direction of an undesired fall to maintain balance. While traditional control engineering methods have been extensively used for bicycle stabilization and path following, there is a growing interest in applying Machine Learning, specifically Reinforcement Learning (RL), to these mechanical systems. A significant gap exists in the literature: previous RL applications to bicycle models have typically specialized in either stabilization (often using simplified models without trail or with balance aids) or path following (often ignoring lateral stabilization). Furthermore, existing works have not sufficiently demonstrated whether an RL agent can induce a speed-dependent stabilization policy for a standard bicycle model (with positive trail) across a wide range of velocities while simultaneously performing path following, without the ability to control forward speed.

Methodology
The authors propose an RL framework where an agent controls a virtual Whipple bicycle model—a benchmark model for bicycle dynamics characterized by positive trail and self-stability only within a specific speed range (4.3–6 m/s). The bicycle is modeled as a multibody system with four rigid bodies (rear wheel, front wheel, rear body, handlebar) and no external stabilization aids (e.g., inertia wheels).

Control Mechanism: The agent acts exclusively by predicting a steering angle ( $\delta_{set}$ ). This angle is converted into a steering torque via a PD controller acting between the rear body and the fork. The agent cannot alter the forward velocity, which is set randomly between 2 m/s and 7 m/s, simulating a scenario where a cyclist controls speed while the agent handles steering.
State Representation: The state vector includes the minimal coordinates of the bicycle (position, yaw, roll, steering angle, and relevant angular velocities) and preview information of the path. The path preview is represented by a vector of lateral distances to future points on the path. The authors investigated two formulations for the state: one using the yaw angle directly and another using a unit vector representation to avoid discontinuities.
Reward Function: The reward is a linear combination of two components:
1. Path Following ( $\rho_y$ ): Rewards the agent for keeping the rear wheel contact point close to the path.
2. Stabilization ( $\rho_\phi$ ): Rewards the agent for maintaining a low roll angle.
  The authors analyzed different reward formulations, including linear vs. exponential distance penalties and varying weights between path following and stabilization.
Training Strategy: Curriculum Learning (CL) is employed to manage the complexity of the task. The training process is divided into four stages where the initialization intervals for forward velocity, path curvature (radius of circular elements), and path complexity (length and width of polynomial segments) are gradually expanded. This allows the agent to first learn basic stabilization at a constant, stable speed before tackling variable speeds and complex paths.
Algorithm: The Soft Actor-Critic (SAC) algorithm is used for off-policy Deep Reinforcement Learning.

Key Contributions

Simultaneous Path Following and Stabilization: The work demonstrates, for the first time, that an RL agent can successfully perform path following and lateral stabilization simultaneously on a Whipple bicycle model without balance aids, relying solely on steering inputs.
Speed-Dependent Policy: The study investigates and confirms that the RL approach can induce a speed-dependent policy capable of stabilizing the bicycle across a wide velocity range (2 m/s to 7 m/s), covering both unstable low-speed and self-stable mid-speed regimes.
Curriculum Learning Application: The application of Curriculum Learning is shown to be essential for training the agent to handle the increasing complexity of variable speeds and arbitrary paths (including full circles, slaloms, and lane changes).
Explainability: The authors utilize SHapley Additive exPlanations (SHAP) to analyze the trained agent's decision-making process, linking the learned policy to known physical mechanisms of bicycle dynamics.

Results

Learning Performance: The proposed settings (using a velocity-scaled preview distance and a linear reward for path following) required fewer learning steps ( $<2 \times 10^6$ ) compared to alternatives like constant preview distance or equal weighting of stabilization rewards. Including the stabilization reward ( $\rho_\phi$ ) with equal weights was found to disrupt learning progress.
Deployment Performance: Agents trained with the proposed settings successfully stabilized the bicycle and followed paths across the entire tested velocity range (2–7 m/s).
- On randomly generated paths, all 525 simulated rides (across 5 runs and 5 settings) were successful with no falls.
- On a complex benchmark path (full circle, slalom, lane changes), the best agent achieved a maximum mean distance of 0.09 m from the path at 7 m/s (9% of the wheelbase) and a maximum mean distance of 0.19 m at the lowest speed (2 m/s).
- Some individual agents failed at the lowest speeds (2 m/s) on specific benchmark segments (full circle and hard lane change), which were statistically rare in the training curriculum.
Mechanism Analysis: SHAP value analysis revealed that the agent learned fundamental control mechanisms:
- Counter-steering: The agent correctly predicted a steering angle opposite to the desired turn direction to initiate a lean.
- Steering into the fall: Once leaning, the agent steered into the direction of the fall to maintain balance, correlating the roll angle ( $\phi$ ) and preview points ( $t_P$ ) with steering actions.

Significance and Claims
The paper claims that Reinforcement Learning is a viable alternative to traditional control engineering for bicycle dynamics, capable of handling the non-linear, speed-dependent nature of the system without explicit modeling of the control laws. The work bridges the gap between machine learning and bicycle dynamics research by demonstrating that an agent can learn the basic physical mechanisms of bicycle control (counter-steering and leaning) purely through interaction with a simulated environment.

The authors modestly note that while the virtual environment allows for perfect path following in many cases, the benchmark path includes geometric features (like hard lane changes) that are physically challenging even for real bicycles. The success of the agent in navigating these complex paths across a wide speed range, despite not having explicit control over forward velocity, underscores the practical potential of RL for robotic systems where velocity is an external constraint. The paper concludes that future work should focus on refining the bicycle model for Sim-to-Real transfer and developing methods to obtain path preview information in real-world scenarios (e.g., via cameras).