💻 computer science

QuadPiPS: A Perception-informed Footstep Planner for Quadrupeds With Semantic Affordance Prediction

This paper introduces QuadPiPS, a perception-informed framework that leverages a novel semantic-geometric "legged egocan" representation and an enhanced ALEF planning strategy to enable safe, terrain-aware, and kinodynamically feasible locomotion for quadrupeds in challenging environments.

Original authors: Max Asselmeier, Ye Zhao, Patricio A. Vela

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Max Asselmeier, Ye Zhao, Patricio A. Vela

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a four-legged dog trying to navigate a chaotic backyard filled with puddles, loose rocks, narrow garden paths, and a steep ramp. If you were a robot, you'd need a brain that doesn't just "see" the ground but understands it.

This paper introduces QuadPiPS, a new "brain" for four-legged robots (quadrupeds) that helps them figure out exactly where to step, even in tricky situations where a simple map would fail.

Here is the breakdown of how it works, using everyday analogies:

1. The Problem: The "Blurry Map" vs. The "High-Res Photo"

Most robots today navigate using Elevation Maps. Think of this like a pixelated, low-resolution satellite photo of the ground. It tells the robot, "There is a hill here," or "There is a hole there."

The Flaw: If you have a small stepping stone surrounded by mud, a pixelated map might blur them together. The robot thinks, "Oh, it's just one big muddy patch," and tries to step right into the mud, slipping and falling.

QuadPiPS uses a different approach called Perception-in-Perception Space. Instead of building a blurry map first, it looks at the raw camera feed like a high-definition photo. It doesn't try to summarize the world; it analyzes the image directly to find safe spots.

2. The "Egocan": The Robot's 360° Bubble

To see the world, the robot uses a tool called an Egocan.

The Analogy: Imagine the robot is wearing a giant, transparent, spherical helmet that wraps around it. As the robot moves, the world gets "painted" onto the inside of this helmet.
The Innovation: Usually, these helmets only show what's in front. QuadPiPS adds "caps" to the top and bottom of the helmet. This means the robot remembers what was directly under its feet a second ago, even after it has walked past it. It creates a continuous, 360-degree bubble of memory.

3. The "Superpixels": Cutting the Cake into Bite-Sized Pieces

Once the robot has its 360° bubble, it needs to decide where to step.

The Old Way: The robot looks at the ground as a grid of tiny squares (like a chessboard). It tries to pick a square.
The QuadPiPS Way: The robot uses an algorithm called Superpixels. Imagine you are looking at a mosaic or a stained-glass window. Instead of looking at every single tiny piece of glass, you group them into natural shapes (like a big blue patch of sky or a jagged red rock).
Why it helps: If there is a single, perfect stepping stone, the Superpixel algorithm sees it as one distinct shape. It doesn't get confused by the noise around it. This allows the robot to say, "That whole blue shape is a safe place to put my foot," rather than guessing pixel by pixel.

4. The "Smart Brain": Geometry + Semantics

QuadPiPS doesn't just look at shape; it looks at meaning.

Geometry: It checks if a surface is flat enough to stand on (like checking if a table is level).
Semantics (The "Affordance"): It uses AI to ask, "Is this thing meant to be stepped on?"
- Example: Imagine a ramp. Geometrically, it's flat. But if it's a ramp leading to a wall, the robot knows, "I can walk up this, but I shouldn't try to stand on the wall at the top."
- QuadPiPS learns to label things as Steppable (Green), Passable (Yellow - I can jump over it), or Non-Passable (Red - Danger!).

5. The "Search & Optimize": Planning the Dance

Once the robot knows the safe shapes, it has to figure out the dance moves.

The Search: It uses a "search graph" (like a maze solver) to find a sequence of safe shapes that get it from Point A to Point B. It's like planning a route across a field of stepping stones, checking every possible combination to find the safest path.
The Optimization: Once the path is chosen, a math engine calculates the perfect swing of the legs to land exactly on those spots without tripping.
The Control: Finally, a "reflex" system (MPC and WBC) watches the robot's body in real-time. If the robot slips a little, this system instantly adjusts the muscles to keep it upright, just like a human instinctively flails their arms to stay on a wobbly bus.

Why This Matters (The Results)

The researchers tested this on two robots: ANYmal C (a research dog) and Unitree Go2 (a consumer robot).

The Test: They threw the robots into 10 different scary scenarios: narrow beams, piles of rubble, ramps, and sparse stepping stones.
The Winner: In easy terrain, everyone did fine. But in hard terrain (like the sparse stepping stones where there are very few places to step), the old "blurry map" robots failed constantly. They couldn't find the small stones.
QuadPiPS succeeded almost every time. Because it could see the stones as distinct, safe shapes, it could plan a path where other robots would just fall off.

The Bottom Line

QuadPiPS is like giving a robot the eyes of a hawk and the brain of a parkour expert. Instead of relying on a blurry, pre-drawn map, it looks at the world in high definition, groups the ground into natural "safe zones," and plans a precise, safe path through chaos. It proves that for robots to walk safely in the real world, they need to understand the texture and meaning of the ground, not just its height.

1. Problem Statement

Legged robots require robust foothold planning to navigate complex, unstructured terrains. Traditional approaches often rely on 2.5D elevation maps or voxel grids, which require computationally expensive point cloud processing (filtering, ray casting) and can struggle with occlusions or confined spaces. Furthermore, many existing planners lack semantic understanding of the terrain (e.g., distinguishing between a surface that is geometrically flat but semantically unsafe to step on).

The core challenge addressed by this work is creating a real-time, perception-informed foothold planning framework that:

Minimizes sensor preprocessing latency.
Integrates both geometric (surface normals) and semantic (steppability) affordances.
Operates effectively in safety-critical environments with sparse or limited footholds (e.g., stepping stones, narrow beams).
Generates kinodynamically feasible whole-body trajectories for quadrupeds.

2. Methodology: The QuadPiPS Framework

QuadPiPS (Quadrupedal Planning in the Perception Space) operates within the Planning in the Perception Space (PiPS) philosophy, acting directly on raw sensor data representations rather than pre-processed maps. The framework consists of four main stages:

A. Perception Pipeline: The Legged Egocan

The system utilizes an extended Egocan representation, a 360° ego-centric model derived from a single front-facing depth camera.

Egocylinder + Egocaps: The standard egocylinder captures the surroundings, while a lower "egocap" captures the floor area directly beneath the robot, effectively acting as a height map without point cloud processing.
Multi-Channel Affordance Encoding: The egocan is augmented with three channels:
1. Depth ( $I_{depth}$ ): Raw distance data.
2. Surface Normals ( $I_{normal}$ ): Estimated via image-space gradients to capture geometric orientation.
3. Steppability Labels ( $I_{step}$ ): Semantic segmentation predicting if a region is Steppable, Passable (can be stepped over), or Non-passable. This is predicted using a DeepLabV3+ model trained on synthetic data generated via primitive shapes (cuboids, ramps, etc.).
Superpixel Oversegmentation: Instead of a uniform grid, the egocan floor is oversegmented into superpixels using a modified SLIC (Simple Linear Iterative Clustering) algorithm. This creates a set of convex planar regions that serve as candidate footholds, allowing the planner to decide where within a feature to step, not just which feature.

B. Motion Planning (Hierarchical Search)

Inspired by the ALEF (Augmented Leafs with Experience on Foliations) framework, the planner decomposes the problem into discrete and continuous spaces:

Torso Search: A coarse $A^*$ search over planar regions generates a guiding torso path.
Contact Mode Family Graph Search: A discrete $A^*$ $A^{*}$ search constructs a graph where nodes represent contact mode families (partial stances) and edges represent transitions between stances.
- Constraints: The search enforces user-defined gait sequences, kinematic reachability (superquadric volumes), stance stability (foot spacing), and swing distance limits.
- Semantic Guidance: The graph search avoids non-steppable regions identified by the semantic model.
Trajectory Optimization (TO): Once a discrete sequence of footholds is found, a sequence of nonlinear trajectory optimization subproblems (solved via the OCS2 library) synthesizes long-horizon, whole-body reference trajectories that transition the robot between stances.

C. Control

The reference trajectory is tracked by a hierarchical control stack:

Model Predictive Control (MPC): Runs at 50 Hz, tracking the reference trajectory while allowing deviations for stability.
Whole Body Control (WBC): Runs at >1 kHz, solving a hierarchical Quadratic Program (QP) to generate joint torques, prioritizing dynamic feasibility over strict trajectory tracking.

3. Key Contributions

Legged Egocan Representation: Extending the egocan to include geometric surface normals and semantic steppability labels, enabling a 360° local environment model from a single camera without point cloud processing.
Synthetic Data Generation: A novel pipeline using primitive shapes to generate synthetic training data for predicting semantic affordances (steppability) in simulation.
Superpixel-Based Foothold Search: Adapting the SLIC algorithm for sparse multi-channel images to oversegment the terrain, facilitating exhaustive search over candidate footholds in complex environments.
Real-Time ALEF Adaptation: Modifying the ALEF framework to support perception-informed, real-time, and kinodynamically feasible planning for quadrupeds.
Comprehensive Validation: Extensive benchmarking on the ANYmal C (simulation) and Unitree Go2 (hardware) across 10 diverse environments and 5 baselines.

4. Experimental Results

Simulation Benchmarking (ANYmal C)

QuadPiPS was compared against five baselines: Elevation Map-MPC (EM-MPC), Elevation Map-RL (EM-RL), Elevation Map-DTC (EM-DTC), Superpixel-MPC (S-MPC), and QuadPiPS.

Sparse/Complex Terrain: In environments with limited footholds (e.g., Sparse Stepping Stones, Ramped Stepping Stones), elevation map baselines failed completely (0% success) because they merged small stones into large, unsafe surfaces. QuadPiPS achieved 8/10 and 9/10 success rates, respectively, by precisely identifying discrete footholds.
Narrow/Obstructed Paths: On Obstructed Balance Beams and Winding Beams, QuadPiPS outperformed others by using semantic labels to avoid stepping on barriers and by adjusting the torso path to follow terrain curvature rather than a straight line.
Robustness: While heuristic planners (S-MPC) struggled with stability in sparse environments, the exhaustive search of QuadPiPS found stable stance configurations, albeit with slightly longer computation times.

Hardware Deployment (Unitree Go2)

The framework was deployed on a Unitree Go2 equipped with a custom onboard compute suite (NVIDIA Jetson Orin NX + Mini PC).

Scenarios: Successfully navigated ramps, stairs, double/triple platforms, and ramped stepping stones.
Performance: The perception pipeline ran at 20 Hz (limited by the semantic segmentation inference, which runs in a separate thread). The system demonstrated the ability to pitch the torso for clearance, recover from slips, and traverse uneven, shifting terrain (e.g., loose boards).
Latency: The image-based approach allowed for fast updates compared to point-cloud-based elevation mapping pipelines.

5. Significance and Conclusion

QuadPiPS represents a significant shift in legged locomotion planning by moving away from heavy point-cloud processing toward image-space perception.

Efficiency: By bypassing point cloud generation, it reduces computational bottlenecks and latency.
Safety: The integration of semantic affordances prevents the robot from attempting to step on geometrically flat but physically unstable or forbidden surfaces.
Adaptability: The framework excels in "safety-critical" scenarios where footholds are scarce or irregular, a domain where traditional elevation maps and heuristic planners often fail.

The work demonstrates that Planning in the Perception Space (PiPS) is viable for complex legged locomotion, offering a robust, deployable solution for autonomous quadrupeds operating in unstructured real-world environments. Future work suggests integrating learned low-level controllers to further enhance robustness against tracking errors.