🤖 AI

Reinforcement Learning on Cost-Constrained Quadrupedal Hardware

This paper demonstrates that deploying reinforcement learning on low-cost quadrupedal hardware can overcome significant sim-to-real gaps caused by actuator delays by employing a forward model of average delay paired with a time-aware neural network that autonomously learns a robust, biologically inspired central pattern generator.

Original authors: Javier C. Weddington, Bence P. Ölveczky, Stephen A. Baccus

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Javier C. Weddington, Bence P. Ölveczky, Stephen A. Baccus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a puppy to walk by shouting instructions from a room far away, connected by a very long, tangled telephone wire. By the time your voice reaches the puppy's ears, it has already tripped over its own paws. By the time it tries to correct its step based on your new shout, it has already fallen again. This is the daily struggle of "Sim-to-Real" robotics, a field where scientists try to teach robots to move by training them in a perfect, instant video game world, only to send them out into the messy, slow, and noisy real world.

The core problem is "latency," or the delay between a command and the action. In expensive, high-tech robots, this delay is tiny—so small it's almost invisible. But in cheap, affordable robots, the delay can be huge, like waiting for a slow internet connection to load a video. When this delay gets too big, the robot can't just react to what it sees right now; it has to guess what will happen next. This turns the robot's brain into a puzzle: it has to figure out how to walk without knowing exactly where its legs are right now, only where they were a moment ago. The big question is: do we need to build expensive, super-fast robots to solve this, or can we teach a cheap robot to be smart enough to handle the lag?

This paper tells the story of a team that decided to try the second option. They took a very cheap, $300 four-legged robot called the Mini Pupper 2, which suffers from a massive 76-millisecond delay in its motors. Instead of trying to fix the hardware or build a complex bridge of math to guess the robot's position, they looked to nature for inspiration. They asked: how do animals walk when their nerves are slow? The answer lies in the spinal cord, which uses a special rhythm generator to keep legs moving even if the brain is slow to react.

The researchers trained different types of "brain" networks (neural networks) to control this cheap robot. They found that a simple, standard brain (called an MLP) failed miserably unless they spent hours manually tuning it with a complex set of rules. However, a more advanced "time-aware" brain (called an LSTM) did something magical on its own. It learned to ignore the noisy, delayed signals from the robot's sensors and instead generated its own internal rhythm, exactly like the Central Pattern Generators (CPGs) found in the spinal cords of vertebrates.

The results were striking. This self-taught rhythm was so strong that the robot could keep walking even when the researchers artificially added another 320 milliseconds of delay on top of the existing lag. While the simple robot's brain fell apart and stopped walking, the time-aware brain kept its steady, trotting pace. The paper suggests that this isn't just a lucky accident; it's a general strategy. When you force a robot to deal with slow, noisy hardware, the smartest solution it can find is to stop trying to react to the present and start generating its own future.

The team also discovered that trying to use the robot's actual, noisy sensor data (like speed and effort) actually made things worse. Because the cheap sensors were so unreliable, the robot walked better when the team told it to pretend those sensors didn't exist and rely entirely on its internal rhythm. They built a two-layer system: a "rhythm keeper" (the LSTM) that handles the long-term timing, and a simple "predictor" (a small neural network) that guesses where the legs should be, replacing the broken real-world sensors.

In the end, the paper proves that you don't need a $10,000 robot to walk well. You just need a cheap robot with a brain that knows how to wait. By mimicking the biological trick of generating an internal rhythm, the researchers closed the gap between the simulation and the real world, showing that sometimes, the best way to handle a slow, noisy world is to stop listening to it and start dancing to your own beat.

Technical Summary: Reinforcement Learning on Cost-Constrained Quadrupedal Hardware

Problem Statement
Deploying reinforcement learning (RL) policies on low-cost robotic platforms is hindered by a significant "sim-to-real" gap caused by transport latencies and noisy motor feedback. While high-end research quadrupeds utilize brushless quasi-direct-drive actuators with low latency (<5 ms) and high control frequencies, affordable platforms like the Mini Pupper 2 ($300) rely on brushed DC position servos. These low-cost actuators introduce severe non-linearities, sluggish response times, and measured transport delays of approximately 76 ms. This delay transforms the locomotion task from a standard Markov Decision Process (MDP) into a Partially Observable Markov Decision Process (POMDP), where the controller receives outdated state information. Furthermore, these platforms often lack velocity and torque feedback, providing only quantized position data, which exacerbates the partial observability. The core challenge is determining whether to address this bottleneck through complex network architectures that learn to handle delays internally or through extensive engineering interventions (e.g., synthetic feedback models) for simpler networks.

Methodology
The authors trained and evaluated four neural network architectures—Multi-Layer Perceptron (MLP), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Transformer—under identical conditions using NVIDIA Isaac Lab and the PPO algorithm.

Simulation Environment: The training environment utilized a calibrated "DelayedPDActuatorCfg" model matching the real hardware's 76 ms transport delay and amplitude performance. The observation vector included base kinematics, joint states, and previous actions.
Training Constraints: All models were trained without curriculum initially. Only the LSTM converged to a stable gait under these conditions. The MLP required a multi-stage curriculum (progressively increasing delay and penalty weights) to converge, while the GRU and Transformer failed to converge within the training budget.
Deployment Strategies:
- MLP: Required a complex, hand-tuned deployment bridge involving a synthetic PD observer to reconstruct missing velocity and effort data. This approach required tuning six or more parameters (stiffness, damping, inertia, delay steps, etc.) and relied on 60% synthetic observations.
- LSTM: Deployed in an open-loop configuration at 25 Hz. Instead of relying on noisy hardware encoders, the system used a learned, stateless MLP servo model to predict joint positions based on action commands. This reduced the reliance on real sensors to only 10% (IMU angular velocity and projected gravity).

Key Results

Emergent Central Pattern Generator (CPG): The LSTM policy, trained end-to-end without explicit oscillator constraints, spontaneously learned a Central Pattern Generator (CPPG). This manifested as a self-sustaining, rhythmic gait (approx. 2.8 Hz) that was robust to observation noise.
Robustness to Latency: The LSTM demonstrated superior robustness to additional delay perturbations. When subjected to extra delays of up to +320 ms, the LSTM maintained a consistent gait amplitude and frequency. In contrast, the MLP's gait collapsed, losing oscillatory structure and amplitude.
Attractor Dynamics: Principal Component Analysis (PCA) of the internal states revealed that the LSTM traces a stable, elliptical limit-cycle attractor across all delay conditions, indicative of an autonomous oscillator. The MLP, conversely, showed distorted, non-oscillatory trajectories that degraded as delay increased.
Observation Ablation: Experiments injecting Gaussian noise into joint-state observations revealed a crossover point at $\sigma \approx 0.2$ rad. Beyond this noise level, zeroing the joint-state channels (ignoring them) yielded better performance than using noisy feedback. The LSTM degraded more gracefully than the MLP, confirming its reliance on an internally generated rhythm rather than noisy external feedback.
Architecture Efficiency: The GRU failed to converge, suggesting that the specific linear shift-register capacity of the LSTM cell state is critical for delay compensation, rather than just general oscillation capability. The Transformer was deemed impractical due to high computational costs and lack of marginal gain over the LSTM for this specific task.

Significance and Claims
The paper posits that for cost-constrained hardware, the sim-to-real bottleneck is best addressed through architectural inductive bias rather than hardware fidelity or extensive engineering compensation.

Biological Parallel: The study demonstrates that time-aware networks, when subjected to biological-like constraints (high latency, noisy feedback), naturally converge to CPG solutions similar to those found in vertebrate spinal cords. This suggests that autonomous rhythm generation may be a general strategy for cost-constrained embodied control.
Dual-Layer Abstraction: The authors propose a two-layer deployment architecture as optimal for this domain: a recurrent policy (LSTM) for long-horizon temporal tasks (delay compensation, gait generation) paired with a stateless predictor (MLP) for short-horizon servo dynamics. This configuration avoids the compounding autoregressive drift seen in fully recurrent (LSTM-LSTM) systems.
Democratization: By showing that robust locomotion can be achieved on a $300 platform with a 76 ms delay using a single tuning parameter (hardware scale), the work lowers the barrier to entry for robotics research, shifting the focus from expensive hardware to intelligent control architectures.

The authors conclude that high-delay hardware induces CPG-like solutions in time-aware networks, offering a principled path to reducing deployment engineering for low-cost robots.

Technical Summary: Reinforcement Learning on Cost-Constrained Quadrupedal Hardware

More like this