⚡ electrical engineering

HuMam: Humanoid Motion Control via End-to-End Deep Reinforcement Learning with Mamba

HuMam is a state-centric end-to-end reinforcement learning framework for humanoid locomotion that leverages a single-layer Mamba encoder to fuse robot states with footstep targets and a phase clock, achieving superior training stability, efficiency, and energy savings compared to feedforward baselines on the JVRC-1 robot.

Original authors: Yinuo Wang, Yuanyang Qi, Jinzhao Zhou, Pengxiang Meng, Xiaowen Tao

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Yinuo Wang, Yuanyang Qi, Jinzhao Zhou, Pengxiang Meng, Xiaowen Tao

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Robot to Walk Without "Overthinking"

Imagine you are trying to teach a robot to walk. The robot has two legs, a torso, and 12 joints in its legs. To walk, it needs to coordinate all of them perfectly while balancing its weight, reacting to the ground, and following a path.

Usually, when we teach robots this, we use a "brain" (a neural network) that looks at the current situation and decides what to do next. The problem is that many of these brains are either too simple (they miss important details) or too heavy and slow (they take too much computer power to run).

The authors of this paper built a new brain called HuMam. The secret sauce? They used a new type of AI architecture called Mamba.

The Analogy: The "Smart Conductor" vs. The "Chaotic Orchestra"

Think of the robot's body as an orchestra with 12 different instruments (the leg joints).

The Old Way (Baseline): The conductor (the AI) looks at the sheet music, but it forgets the notes it just played. It tries to guess the next note based only on the current moment. Sometimes it gets the rhythm wrong, the orchestra stumbles, and the robot falls over.
The HuMam Way: The conductor uses a Mamba system. This isn't just a conductor; it's a conductor with a perfect, instant memory of the flow of the music. It doesn't need to re-read the whole song from the beginning every time. It knows exactly how the last note connects to the next one. This allows the robot to move smoothly, like a dancer, rather than a clumsy walker.

What Makes HuMam Special?

1. The "Lightweight" Brain (Mamba)

The paper claims this is the first time a Mamba architecture has been used to control a humanoid robot's walking.

The Metaphor: Imagine trying to carry a heavy backpack (a complex AI model) while running a marathon. It slows you down. HuMam uses a "Mamba" layer, which is like a feather-light, high-tech backpack. It holds all the necessary information but weighs almost nothing. This means the robot can think faster and use less battery power.

2. The "Six-Point Scorecard" (Reward Shaping)

To teach the robot, the computer gives it points (rewards) for doing things right. The authors created a specific "scorecard" with six items:

Don't stomp: Keep your feet gentle on the ground.
Swing smoothly: Don't kick your legs wildly.
Hit the target: Step exactly where you planned to.
Stand tall: Keep your head up and back straight.
Stay at the right height: Don't crouch too low or stand too tall.
Don't wobble: Keep your upper body steady.

The robot tries to get the highest possible score on this card. Because the Mamba brain is so good at connecting these dots, the robot learns to balance all six goals at once without sacrificing one for the other.

The Results: Faster, Smoother, and Greener

The researchers tested HuMam on a virtual robot named JVRC-1 in a computer simulation. They compared it to a standard robot brain (the Baseline) that didn't use Mamba. Here is what happened:

Faster Learning: HuMam learned to walk much faster. It needed 13% to 42% fewer practice attempts to reach the same skill level as the other robot.
- Analogy: If the old robot needed to practice walking for 100 hours to get good, HuMam only needed 60 hours.
More Stable: The old robot's learning was "jittery"—sometimes it would get really good, then suddenly get worse. HuMam was consistent.
- Analogy: The old robot was like a student who aced a test one day and failed the next. HuMam was the student who got an A every single time.
Energy Efficient: This is a big deal. HuMam used less electricity to walk the same distance.
- Analogy: The old robot was like a gas-guzzling SUV that revs its engine loudly. HuMam is like a hybrid car that glides silently. In the "Standing" test, HuMam used 39% less power just to stay upright.
Smoother Joints: The robot's joints (hips, knees, ankles) moved with less jerky force.
- Analogy: The old robot's joints sounded like a rusty hinge; HuMam's joints sounded like a well-oiled door.

The Bottom Line

The paper argues that by using this new "Mamba" brain, they created a robot controller that is:

Lighter (uses less computer power).
Smarter (learns faster and more consistently).
Greener (uses less energy).

They successfully proved that you don't need a massive, heavy computer brain to make a robot walk well. Sometimes, a lightweight, efficient "Mamba" brain is all you need to get a robot to dance across the room.

Note: The paper focuses entirely on computer simulations (virtual robots). It does not claim to have tested this on a real, physical robot yet, though it mentions that is a goal for future work.

Technical Summary: HuMam – Humanoid Motion Control via End-to-End Deep Reinforcement Learning with Mamba

1. Problem Statement

Humanoid locomotion presents unique challenges compared to quadrupedal or wheeled systems, primarily due to higher centers of gravity, coupled upper-lower body dynamics, and strict safety requirements. While end-to-end reinforcement learning (RL) offers a compact perception-action mapping, practical deployment often suffers from training instability, inefficient feature fusion, and high actuation costs.

Existing approaches face specific limitations:

Feedforward backbones often under-exploit structured observation data, such as gait phase and oriented footstep targets.
Heavy sequence models (e.g., Transformers or deep RNNs) impose unfavorable compute-memory trade-offs, hindering real-time onboard deployment.
Training dynamics frequently exhibit instability, high variance across seeds, and inefficient sample usage, leading to policies that are either unstable or energy-inefficient.

The paper addresses the need for a controller that is both foresightful (coordinating foot placement and balance) and resource-aware (operating within onboard compute and actuation limits) without relying on explicit temporal history accumulation.

2. Methodology: The HuMam Framework

The authors propose HuMam, a state-centric, end-to-end RL framework designed for humanoid walking. The core innovation lies in the adoption of a single-layer Mamba encoder as a lightweight fusion backbone.

2.1. Observation Space

The agent receives a rich set of observations at each time step, divided into two categories:

Robot-centric states: Joint positions and velocities for 12 actuated leg joints, root orientation (roll, pitch), and angular velocity (roll, pitch, yaw). Global yaw is excluded as footsteps are defined relative to the root frame.
External guidance: Two oriented footstep targets ( $T_1, T_2$ ) defined by 3D position and heading, plus a continuous phase clock encoding the gait cycle ( $\sin, \cos$ ) to prevent discontinuities associated with scalar phase encoding.

2.2. Architecture: Single-Layer Mamba Encoder

Unlike recurrent or attention-based designs that accumulate temporal history, HuMam relies solely on the current state but utilizes Mamba's selective state-space dynamics to capture structured dependencies between input modalities.

Tokenization: Robot and external states are concatenated and projected into modality-specific embeddings.
State-Space Encoding: A single Mamba layer applies gated state-space dynamics to these tokens. This formulation provides a convolutional view with exponentially decaying kernels, enabling efficient feature mixing without explicit temporal recurrence.
Output: The encoded representation is aggregated into a compact feature vector shared by the policy and value heads.

2.3. Control Hierarchy and Action Space

Action Space: The high-level policy (running at 40 Hz) outputs desired joint position targets for the 12 actuated leg joints.
Low-Level Execution: A low-gain Proportional-Derivative (PD) controller (running at 1000 Hz) converts these position targets into executable joint torques. This hierarchical structure ensures smooth torque execution and keeps the learning problem well-conditioned.

2.4. Reward Shaping

The framework employs a concise six-term reward function optimized via Proximal Policy Optimization (PPO) to balance stability, accuracy, and energy efficiency:

Foot Force ( $R_{force}$ ): Penalizes excessive ground reaction forces.
Foot Velocity ( $R_{vel}$ ): Encourages smooth swing leg motions.
Step Accuracy ( $R_{step}$ ): Rewards precise foot placement relative to targets.
Orientation ( $R_{orient}$ ): Aligns the root orientation with the desired heading.
Body Height ( $R_{height}$ ): Maintains a nominal torso height.
Upper-Body Stability ( $R_{upper}$ ): Minimizes upper-body sway.

3. Key Contributions

The paper outlines four primary contributions:

First Mamba-backed Humanoid RL Controller: Introduces Mamba as a lightweight fusion backbone for end-to-end humanoid locomotion, demonstrating that a single layer suffices for strong performance in a state-centric setting.
Backbone Effectiveness: Shows that the Mamba encoder yields faster learning, improved sample efficiency, and reduced cross-seed variability compared to an identically trained feedforward baseline.
Energy-Aware Reward Shaping: Proposes a six-term reward that jointly regulates contact forces, swing velocities, stepping accuracy, and body stability, implicitly promoting energy efficiency.
Extensive Empirical Validation: Validates the framework on the JVRC-1 humanoid across diverse tasks (forward, backward, curved, lateral walking, and standing) in the mc-mujoco environment.

4. Experimental Results

Experiments were conducted on the JVRC-1 humanoid (1.72m, 62kg) using mc-mujoco. HuMam was compared against a strong feedforward baseline under identical training settings.

4.1. Learning Efficiency and Stability

Sample Efficiency: HuMam required 15.8M samples to reach a return of 240, compared to 18.2M for the baseline (13.2% improvement). At a higher threshold (return 260), the improvement was 42.5% (20.2M vs. 35.0M samples).
Stability: HuMam reduced cross-seed learning curve deviation by 35.4% and late-stage training variance by 61.0%, indicating more reliable convergence.
Performance: Peak performance increased by 5.8% (269.85 to 285.50), and the mean return over the final 10% of training improved by 5.5%.

4.2. Reward and Torque Profiles

Reward Components: HuMam achieved a higher average per-step reward (0.737 vs. 0.728), with notable improvements in foot force regulation and velocity smoothness.
Torque Reduction: The framework reduced average joint torque magnitudes by 9.6% and peak torques by 9.1%. Specific joints, such as the left hip pitch and right ankle pitch, saw significant reductions (up to 40% in average torque for the ankle).

4.3. Energy Efficiency

HuMam demonstrated superior energy economy across dynamic tasks:

Standing: Reduced average power consumption from 57.48 W to 35.08 W (39% decrease).
Forward Walking: Reduced energy cost from 673 J/m to 421 J/m.
Lateral Walking: Reduced energy cost from 1395 J/m to 963 J/m.
Power per Mass: Consistently lower W/kg across all tasks, with an average reduction from 1.92 W/kg (baseline) to 1.38 W/kg (HuMam).

5. Significance and Claims

The authors claim that HuMam represents the first end-to-end humanoid RL controller to adopt Mamba as a fusion backbone. The paper argues that this architecture provides tangible gains in three critical areas:

Efficiency: Achieving higher performance with fewer training samples and lower computational overhead due to the near-linear complexity of the single-layer Mamba encoder.
Stability: Reducing training variance and producing smoother, more robust gaits across diverse locomotion scenarios.
Control Economy: Generating policies that require less torque and energy, making them more suitable for real-world deployment on hardware with limited power budgets.

The work concludes that the Mamba backbone is an effective encoder for compact, state-centric policy learning, enabling the synthesis of natural and stable humanoid walking behaviors without relying on explicit temporal history or heavy sequence models. Future work is directed toward hardware deployment, sim-to-real transfer, and integrating onboard perception.