A Hybrid Autoencoder for Robust Heightmap Generation from Fused Lidar and Depth Data for Humanoid Robot Locomotion

Technical Summary: A Hybrid Autoencoder for Robust Heightmap Generation

Problem Statement
Reliable locomotion for humanoid robots in unstructured, human-centric environments is hindered by the inherent instability of bipedal morphology and the limitations of current perception systems. Traditional approaches often rely on single-sensor pipelines (either depth cameras or LiDAR) or computationally expensive, manually engineered mapping techniques like SLAM and voxelization. These methods frequently suffer from sensitivity to lighting, surface properties, significant latency, and drift. Furthermore, existing perception-to-control pipelines often lack a robust intermediate representation that balances terrain detail with the computational efficiency required for real-time control.

Methodology
The paper proposes a learning-based, modular framework that decouples perception from control, utilizing a robot-centric heightmap as the intermediate representation. The core of the system is a Hybrid Encoder-Decoder Structure (EDS) designed to reconstruct terrain from raw, high-dimensional sensor data.

• Sensor Fusion: The system fuses data from three sources: an Intel RealSense depth camera, a LIVOX MID-360 LiDAR, and an onboard IMU.
  • LiDAR Processing: To enable efficient CNN processing, the unstructured 3D LiDAR point cloud is transformed via spherical projection into a structured 2D range image (276 × 40 resolution). This includes preprocessing steps like gap-filling, nearest-neighbor interpolation, and median filtering to mitigate projection artifacts.
  • Depth Processing: Depth images are processed at 160 × 120 resolution.
• Architecture:
  • Encoders: Separate Convolutional Neural Network (CNN) branches process the depth and LiDAR range images. Each branch uses four strided convolutional layers to downsample spatial resolution and extract features, mapping them to a 256-dimensional latent vector.
  • Fusion & Temporal Core: The latent vectors are concatenated with a 15-dimensional robot state vector (IMU data) and the heightmap prediction from the previous timestep. This unified representation is processed by a recurrent core consisting of two stacked Gated Recurrent Unit (GRU) layers (256 hidden units) to ensure temporal consistency and handle occlusions.
  • Decoder: A fully connected decoder head outputs the final 165-dimensional predicted heightmap.
• Training Strategy: A two-stage training process is employed. First, symmetric convolutional autoencoders are pre-trained in an unsupervised manner on noisy simulated data to stabilize feature extraction. Second, these encoders are integrated into the full EDS with the GRU core and trained in a supervised setting using ground-truth heightmaps.
• Heightmap Optimization: The robot-centric heightmap is optimized for locomotion policy behavior. The study identifies an optimal grid resolution of 7 cm within a 0.98 m × 0.7 m grid, with the center shifted 0.2 m forward to prioritize the robot's immediate path.
• Control Policy: A Deep Reinforcement Learning (DRL) policy, trained via Proximal Policy Optimization (PPO) in Isaac Lab, utilizes the generated heightmap, proprioceptive signals, and velocity commands to control 23 actuated degrees of freedom. The reward function is shaped to encourage anticipatory behavior, such as lifting feet for upcoming elevation changes.

Key Contributions

1. Hybrid EDS Architecture: Introduction of a multimodal fusion architecture combining CNNs for spatial feature extraction and GRUs for temporal modeling, specifically tailored for humanoid terrain perception.
2. Optimized Heightmap Representation: Systematic evaluation of grid spacing and resolution, establishing that a 7 cm resolution with a forward-shifted field of view provides the optimal trade-off between foothold precision and training stability.
3. Efficient LiDAR Projection: Implementation of a spherical projection method for LiDAR data that converts 3D point clouds into 2D range images suitable for CNNs, reducing computational complexity compared to voxel-based methods.
4. Two-Stage Training: A methodology separating unsupervised pretraining of encoders from supervised temporal fusion to improve convergence and robustness.

Results

• Perception Accuracy: The multimodal fusion approach achieved a Mean Absolute Error (MAE) of 2.19 cm. This represents a 7.2% improvement over depth-only configurations (2.36 cm) and a 9.9% improvement over LiDAR-only configurations (2.43 cm).
• Temporal Context: Utilizing a temporal horizon of 3.2 seconds (32 timesteps) reduced reconstruction errors by approximately 30% compared to shorter horizons, with diminishing returns observed at 6.4 seconds.
• Locomotion Stability: The integration of the optimized heightmap and reward shaping resulted in a 70.1% reduction in terminations caused by falls compared to a blind baseline. Additionally, linear velocity error decreased by 25.0% and angular velocity error by 17.2%.
• Robustness: The policy maintained stable locomotion even when heightmap inputs were corrupted with Gaussian noise up to a standard deviation of 2 cm.

Significance and Limitations
The paper claims to establish a validated foundation for real-time, terrain-aware humanoid locomotion by decoupling perception from control and optimizing the intermediate representation. The framework demonstrates that hybrid sensor fusion and tailored grid spacing enable anticipatory gait patterns, allowing robots to navigate complex terrains more safely.

However, the authors acknowledge specific limitations:

• Smoothing of Discontinuities: The pixel-wise Mean Squared Error (MSE) loss function tends to smooth sharp vertical discontinuities (e.g., stair edges) into gradual slopes, which may reduce foothold precision on discontinuous surfaces.
• Noise Sensitivity: Control reliability degrades significantly when heightmap noise exceeds a 2 cm standard deviation.
• Representation Limits: The 2.5-D heightmap cannot capture overhanging structures.
• Sim-to-Real Gap: While the system performs well in simulation, a gap remains for real-world deployment, particularly regarding pretraining on cluttered real-world datasets.

The authors suggest that future work should focus on edge-preserving loss functions, hybrid CNN-Transformer architectures, and comprehensive domain randomization to address these limitations.