💻 computer science

Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System

This paper demonstrates successful zero-shot transfer of reinforcement learning policies for cart-pole swing-up and stabilization from simulation to hardware by combining bandwidth-aware filtering, sensitivity-guided domain randomization, and a linear curriculum learning schedule to ensure robust energy injection and disturbance rejection.

Original authors: Nikki Xu, Hien Tran

Published 2026-06-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikki Xu, Hien Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to balance a broomstick on its hand. This is the "Cart-Pole" problem: a cart moves back and forth on a track, and a pole is hinged to it. The robot has two jobs:

Swing-Up: Get the pole from hanging straight down to standing straight up.
Stabilization: Keep it standing up without falling over.

The researchers at NC State University wanted to teach a robot to do this using Reinforcement Learning (RL). Think of RL like training a dog: you let the dog try things, and if it does something good, you give it a treat (reward). If it messes up, it gets no treat. Eventually, the dog learns the best tricks.

However, there is a big problem: The "Sim-to-Real" Gap.
You can train a robot perfectly in a computer simulation (a video game world), but when you put that same "brain" into a real robot, it often fails. Why? Because the real world has friction, wobbly sensors, and imperfect motors that the computer simulation didn't account for. It's like training a swimmer in a calm, perfect pool and then dropping them into a choppy ocean; they might drown because the conditions are different.

The Problem: The Robot Broke Itself

In the beginning, the researchers tried to train a robot to swing the pole up using a standard computer simulation.

The Result: The computer thought the robot was doing great. But when they tried it on the real machine, the robot went crazy.
The Analogy: Imagine a driver who learned to drive in a video game. In the game, they can slam the gas pedal from 0 to 100 instantly. In real life, if you do that, your tires spin, you lose control, and you might break the car.
What happened: The "brain" the computer learned told the real motor to switch from full speed forward to full speed backward in a split second. This violent shaking broke the plastic wheels of the cart. The robot was too "jittery."

The Solution: A Three-Step Recipe

To fix this, the researchers developed a specific recipe to make the computer-trained brain work on the real machine without any extra tuning (this is called "Zero-Shot Transfer," meaning it works immediately, like a plug-and-play device).

1. The "Smoothie" Filter (Action Smoothing)

The Issue: The computer brain was giving orders that were too jerky and fast, like a drum solo played at 1,000 beats per minute.
The Fix: They added a "low-pass filter." Think of this as a smoothie blender for the robot's commands. If the brain wants to scream "GO LEFT!" and then immediately "GO RIGHT!", the blender smoothes that out into a gentle "Go left, then slowly turn right."
Why it matters: This protected the physical wheels from breaking and stopped the robot from shaking itself apart.

2. The "Training Wheels" (Curriculum Learning)

The Issue: If you throw a student into a hard exam immediately, they might panic and fail.
The Fix: They used a "Curriculum." Instead of training the robot in a perfect, easy world, they started with a slightly messy world and slowly made it messier.
The Analogy: It's like learning to ride a bike. You start on flat ground with training wheels. Once you get good, you take the wheels off. Then you practice on a slight hill. Finally, you practice on a bumpy road. By the time the robot is "graduated," it has seen so many different types of "bumps" in the simulation that the real world feels easy.

3. The "Targeted" Randomization (Sensitivity-Guided Domain Randomization)

The Issue: If you try to randomize everything (the weight of the cart, the wind, the friction, the color of the sky), the robot gets confused and learns nothing.
The Fix: The researchers did a "sensitivity analysis." This is like a doctor checking which parts of the body are most sensitive to a disease. They found that only three specific parts of the robot's motor (how heavy the spinning part is, and two electrical constants) really mattered.
The Strategy: They only randomized those three specific parts during training. They didn't mess with everything else. This kept the training focused but robust enough to handle the real world.

The Final Act: The Handoff

The robot has two different "brains" (policies): one for swinging the pole up, and one for keeping it balanced.

The Challenge: You can't just switch from the "Swing-Up Brain" to the "Balance Brain" instantly. If you switch at the wrong moment, the pole will fall.
The Solution: They created a "Switching Logic." Imagine a traffic light. The "Swing-Up Brain" drives the car until the pole is almost vertical and the cart is in the right spot. Then, the traffic light turns green, and the "Balance Brain" takes over. They made sure this switch was smooth (no "bumps") so the robot didn't get startled.

The Result

What worked: The combination of the Smoothie Filter (to stop breaking), Targeted Randomization (to learn the right things), and Curriculum Learning (to learn gradually) allowed the robot to go from a hanging position to a standing position and stay there.
What didn't work: If they randomized too many things, or didn't use the filter, the robot failed in the real lab, even if it looked perfect in the computer.
Key Takeaway: You can't just train a robot in a perfect video game and expect it to work in the messy real world. You have to teach it to handle "noise" and "jerkiness" while it's learning, using a gentle, step-by-step approach.

In short, they built a robot that learned to balance a pole by training it in a messy, simulated classroom, smoothing out its commands so it wouldn't break its own legs, and letting it switch to a "balance mode" only when it was ready. And it worked immediately when they turned it on in the lab.

Technical Summary: Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System

Problem Statement
The paper addresses the challenge of deploying Reinforcement Learning (RL) control policies from simulation to physical hardware without fine-tuning (zero-shot transfer). Specifically, it targets the cart-pole system, a canonical nonlinear control benchmark requiring two distinct phases: a "swing-up" phase to inject energy and move the pendulum from a downward equilibrium to an upright region, and a "stabilization" phase to maintain balance against disturbances. While RL has shown success in simulation, direct transfer to hardware often fails due to the "sim-to-real gap," caused by model mismatches, sensor noise, and unmodeled dynamics. Previous attempts in this work using high-fidelity models without specific transfer techniques resulted in hardware damage due to high-frequency, "bang-bang" control signals.

Methodology
The authors propose a hybrid control framework involving two independently trained policies (swing-up and stabilization) managed by a switching logic in Simulink. The core methodology focuses on three specific techniques to bridge the sim-to-real gap without post-transfer fine-tuning:

Action Smoothing (Low-Pass Filtering): To prevent hardware damage from high-frequency oscillatory actuation inherent in discrete-time RL optimization, a first-order low-pass filter is applied to the neural network's output signal. This acts as a bandwidth-aware filter to smooth control inputs.
Sensitivity-Guided Domain Randomization (DR): Instead of randomizing all model parameters indiscriminately, the authors perform a sensitivity analysis using a complex-step method to identify parameters that significantly influence system dynamics. Randomization is then applied only to a subset of parameters (specifically motor moment of inertia, torque constant, and back-EMF constant) to reduce overfitting to the nominal model while preserving system dynamics.
Curriculum Learning (CL): A linear curriculum schedule is implemented where the uncertainty margins for randomized parameters gradually increase from zero to a maximum value over a fixed number of training episodes. This aims to stabilize initial training and improve final performance.
Policy Switching: A switching logic is designed to transition between the swing-up and stabilization policies. The authors implement hysteresis-like behavior and relaxed state bounds to ensure smooth handoffs and avoid chattering, allowing the swing-up policy to re-engage if the pendulum falls.

Key Results
The study compares several training configurations through simulation and physical hardware experiments:

Initial Failure: A controller trained without transfer techniques (no DR, no CL, no filtering) succeeded in simulation but catastrophically failed on hardware within seconds, breaking the cart's plastic wheels due to violent voltage switching.
Case 0 (Indiscriminate Randomization): Randomizing all physical parameters with a 10% margin, even with curriculum learning, resulted in blurred performance benefits and did not consistently outperform non-randomized baselines in simulation.
Case 1 (Targeted Randomization): The most successful configuration involved randomizing only three low-to-moderately sensitive parameters ( $J_m$ , $K_t$ , $K_m$ ) with a 5% margin over 100 episodes, combined with curriculum learning and action filtering. This policy achieved a 100% success rate in simulations under severe disturbance levels.
Hardware Deployment: Only the policy from Case 1 successfully transferred to the physical cart-pole system. It reliably injected sufficient energy to hand off to the stabilizer, maintained balance against manual taps, and recovered from larger perturbations.
Counter-Intuitive Finding: Policies that achieved higher cumulative rewards in simulation (often those with broader randomization or no curriculum) frequently failed to transfer to the hardware. This suggests that simulated reward metrics alone are insufficient indicators of hardware viability.

Significance and Claims
The paper claims to provide a practical pathway for deploying RL-based controllers on physical hardware without the need for fine-tuning. Its primary contributions are:

Empirical Evidence for Zero-Shot Transfer: Demonstrating that a combination of action filtering, sensitivity-guided domain randomization, and curriculum learning can yield a robust controller that works directly on hardware.
Safety-Critical Filtering: Highlighting that action smoothing is not merely a performance enhancer but a safety-critical component necessary to prevent physical damage from RL-induced high-frequency inputs.
Parameter Selection Strategy: Showing that targeted randomization based on sensitivity analysis is more effective than broad, indiscriminate randomization for this specific system, which operates in a relatively stable environment.
Evaluation Protocol: Proposing a comprehensive evaluation metric that includes swing-up time, success rates, and standard deviation of rewards, arguing that mean episodic reward is a poor predictor of real-world performance.

The authors maintain a modest tone, acknowledging that their findings are largely empirical and that the theoretical implications of these techniques for RL controllers require further rigorous investigation. They conclude that while their specific combination of techniques worked for the cart-pole system, the results should be viewed as empirical evidence from a limited campaign rather than a definitive universal rule.