💻 computer science

No More Marching: Learning Humanoid Locomotion for Short-Range SE(2) Targets

This paper presents a reinforcement learning approach with a novel constellation-based reward function that enables humanoids to efficiently and robustly reach short-range SE(2) target poses, outperforming traditional velocity-tracking methods in both simulation and real-world hardware.

Original authors: Pranay Dugar, Mohitvishnu S. Gadde, Jonah Siekmann, Yesh Godse, Aayam Shrestha, Alan Fern

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pranay Dugar, Mohitvishnu S. Gadde, Jonah Siekmann, Yesh Godse, Aayam Shrestha, Alan Fern

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk. For a long time, the best way to teach robots to walk was to tell them, "Walk forward at 1 meter per second." This works great if the robot needs to cross a whole field. But what if the robot just needs to take two steps to the left to grab a cup of coffee?

If you tell a robot to "walk forward" when it only needs to move a few inches, it ends up doing something silly: it marches in place, stops, turns like a tank, marches again, and stops. It's inefficient, clunky, and wastes a lot of energy. It's like trying to park a car by driving forward, stopping, turning the wheel 90 degrees, driving backward, and repeating that until you fit in the spot.

This paper introduces a new way to teach humanoid robots (like the "Digit" robot) how to move short distances efficiently. Instead of giving them marching orders, the researchers taught the robot to aim for a specific spot and angle, just like a human does when they walk to a chair.

Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Marching" Robot

Most robots are trained to follow a speed command (like a car's cruise control). When you ask them to go to a specific spot nearby, they don't know how to stop or turn smoothly. They just keep "marching" until they overshoot, then they have to back up and turn.

The Analogy: Imagine a soldier in a parade. If you tell them to move to a spot 3 feet away, they might take three giant, rhythmic steps, overshoot, and then have to pivot awkwardly. It's rigid and unnatural.

2. The Solution: The "Constellation" Reward

The researchers used a technique called Reinforcement Learning (trial and error with rewards). The secret sauce was how they designed the "reward" the robot gets for doing a good job.

Usually, you might reward a robot for getting the position right and separately for getting the direction right. But the researchers found that balancing these two is like trying to tune a radio while driving a car; it's too complicated and leads to bad habits.

Instead, they invented the Constellation Reward.

The Analogy: Imagine the robot has a glowing, invisible "halo" of dots floating around it (like a constellation of stars). The goal also has a matching "halo" of dots.
How it works: The robot doesn't get a reward for just moving forward or just turning. It gets a reward based on how well its "halo" matches the goal's "halo."
- If the robot is too far away, the dots don't line up.
- If the robot is facing the wrong way, the dots don't line up.
- If the robot is in the right spot but facing the wrong way, the dots still don't line up.

This forces the robot to figure out the best way to move its feet to make the dots match perfectly. It naturally learns to turn while it walks, rather than stopping to turn. It's like a dancer who moves their feet and spins their body simultaneously to reach a pose, rather than walking to a spot and then spinning in place.

3. The Result: The "GoTo" Controller

They trained a robot named Digit using this new method.

In Simulation (Video Game): The robot learned to walk to targets in half the time, using half the energy, and taking fewer steps than the old "marching" methods. It looked fluid and human-like.
In the Real World: They put the code on a real robot. Even though real floors are slippery and sensors are noisy, the robot worked almost as well as in the video game. It successfully walked to targets, turned, and stopped without falling over or doing the awkward "march-turn-march" dance.

Why This Matters

This paper solves a very practical problem. Real-world robots (in warehouses, homes, or factories) rarely need to walk across a field. They mostly need to shuffle a few feet to pick up a box, turn, and put it down.

By changing the "reward" from "keep walking" to "match the constellation," the robot learned to be agile. It stopped acting like a marching soldier and started acting like a person who just wants to get to the fridge quickly and efficiently.

In short: They stopped teaching robots to march and started teaching them to aim, resulting in a robot that moves with the natural, fluid efficiency of a human.

1. Problem Statement

Humanoid robots operating in real-world environments frequently need to perform short-range, task-driven movements to reach specific SE(2) target poses (position $x, y$ and orientation $\theta$ ).

The Challenge: Existing locomotion controllers, particularly those based on Reinforcement Learning (RL), are typically optimized for velocity-tracking (long-range, rhythmic gait generation). When applied to short-range tasks, these controllers produce inefficient, unnatural "marching-style" behaviors (e.g., stopping to rotate in place, then walking forward).
The Gap: There is a lack of control strategies that directly optimize for reaching a specific pose efficiently, balancing the trade-off between translational progress and rotational alignment without rigid, hand-tuned phase separation (e.g., "orient-then-move").

2. Methodology

The authors propose an end-to-end Reinforcement Learning (RL) framework to train a "GoTo" controller that directly maps SE(2) target offsets to joint actions.

A. Core Innovation: Constellation Reward

The central contribution is a novel constellation-based reward function ( $r_{con}$ ) designed to unify translation and rotation objectives.

Concept: Instead of treating position and orientation as separate, weighted terms, the robot's base is represented by a virtual 2D point constellation (a set of $N$ points rigidly attached to the robot).
Mechanism: The reward is derived from the geometric similarity (Mean Squared Distance) between the robot's current constellation and the target constellation.
- $d_{con} = \frac{1}{N} \sum \|p_i - p^*_i\|^2$
Decomposition: The distance metric naturally decomposes into a translational error ( $d_p$ $d_{p}$ ) and a rotational error ( $d_o$ $d_{o}$ ) scaled by the moment of inertia ( $I_c$ $I_{c}$ ) of the constellation:
- $d_{con} \approx d_p + I_c \cdot d_o$
Benefit: This creates a multiplicative coupling between position and orientation errors. By adjusting the spatial extent of the constellation (radius), the system implicitly balances the trade-off between reaching the target location and aligning the orientation, eliminating the need for manual tuning of relative weights.
Reward Function: The final reward is $r_{con} = e^{-w_c \cdot d_{con}}$ , encouraging smooth, coordinated reduction of both errors.

B. Additional Reward Components

Style Reward ( $r_{sty}$ ): Encourages natural bipedal gaits, appropriate base height, foot airtime, and arm positioning (adapted from prior sim-to-real work).
Regularization Reward ( $r_{reg}$ ): Penalizes high torque, excessive joint acceleration, and action changes to ensure energy efficiency and hardware safety.

C. Architecture and Training

Policy: A two-layer LSTM (128, 128) network.
Inputs: Proprioceptive state (joint positions/velocities, root pose) and the delta command ( $\Delta x, \Delta y, \Delta \theta$ ) representing the difference between current and target pose.
Outputs: Joint-space PD setpoints for 20 actuators.
Training: Proximal Policy Optimization (PPO) with mirror loss for symmetry, trained in NVIDIA IsaacLab on the Agility Digit-V3 model.
Sim-to-Real: Extensive domain randomization (joint stiffness, mass, friction, noise, PD gains) is used to ensure robust transfer to hardware.

3. Key Contributions

Constellation Reward: A unified, geometrically grounded reward function that naturally couples translation and rotation, enabling efficient SE(2) target reaching without "marching" behaviors.
End-to-End Learning: A direct policy mapping SE(2) goals to actions, removing the need for intermediate velocity commands or explicit footstep planning.
Benchmarking Framework: A rigorous evaluation suite measuring Position Error, Orientation Error, Time-to-Target, Footstep Count, and Energy Efficiency across a distribution of SE(2) goals.
Sim-to-Real Validation: Successful deployment on the physical Digit v3 humanoid, demonstrating that the learned policy transfers effectively without fine-tuning.

4. Results

The proposed GoTo controller was evaluated against three baselines:

Finite State Machine (FSM): Hand-tuned "orient-then-move" strategy.
Agility Model-Based Controllers: Manufacturer-provided planners (Agility-2 and Agility-3).
Hierarchical Controller: Learned high-level policy outputting velocities for a pre-trained low-level controller.

Quantitative Findings (Normalized against Agility-2):

Energy Efficiency: GoTo consumed 37% less energy (0.63 vs 1.0).
Time-to-Target: GoTo was 44% faster (0.56s vs 1.0s).
Footstep Count: GoTo took 33% fewer steps (0.67 vs 1.0).
Accuracy: GoTo achieved 40% better position accuracy and 58% better orientation accuracy.
Robustness: GoTo showed lower standard deviations across all metrics, indicating higher reliability.

Qualitative Findings:

Baselines (FSM, Agility) exhibited rigid, segmented behaviors (e.g., stopping to turn, then walking).
The GoTo controller generated fluid, human-like locomotion where turning and translation occurred simultaneously and organically, integrating directional changes into forward motion.

Real-World Performance:

Achieved a 90.23% success rate on the physical Digit v3 robot across 72 distinct short-range tasks.
Performance trends in the real world closely mirrored simulation results, confirming the effectiveness of the reward design and domain randomization.

5. Significance

This work demonstrates that task-specific reward design is critical for practical humanoid control. By moving away from velocity-tracking paradigms and adopting a geometric, pose-centric reward (Constellation Reward), the authors achieved:

Elimination of "Marching": The robot learns to navigate naturally rather than following rigid, inefficient phases.
Efficiency: Significant reductions in energy and time are crucial for battery-operated robots in real-world workflows (e.g., logistics, box moving).
Scalability: The approach proves that end-to-end RL can successfully bridge the sim-to-real gap for complex, short-range navigation tasks on non-trivial humanoid morphologies (like the Digit's closed kinematic loops).

The paper establishes a new standard for short-range humanoid locomotion, suggesting that future systems should prioritize direct pose-reaching objectives over velocity tracking for confined, task-driven environments.