💻 computer science

Perception-and-action system for humanoid robot task execution in construction

This paper proposes a novel perception-and-action system comprising Humanoid-PoseNet and Humanoid-ActionNet that enables humanoid robots to learn and reliably execute construction tasks from human demonstrations, achieving an average motion-tracking error of 82.45 mm.

Original authors: Yanxi Liu, Yizhi Liu

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanxi Liu, Yizhi Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just clunky machines on wheels or arms bolted to a wall, but partners that look and move like us. This is the realm of humanoid robotics, a field trying to build machines that can walk into our human-made spaces—like construction sites, kitchens, or offices—and do the heavy lifting without needing a manual for every single step. The big challenge? Humans are messy, flexible, and unique; robots are precise, rigid, and built to specific blueprints. If you try to copy a human's movement directly onto a robot, the robot might trip, fall, or snap a joint because its legs are shorter or its hips bend differently. To solve this, scientists use motion retargeting (a fancy way of saying "translating human moves into robot moves") and reinforcement learning (a method where a robot learns by trial and error, getting "rewards" for staying upright and "punishments" for falling). The goal is to let robots learn from watching us, so they can eventually help us build our world, carrying bricks or climbing ladders just like a human worker would.

Enter this new study, which acts like a master translator and a tough coach for a robot named Unitree G1. The researchers wanted to see if a humanoid robot could watch a construction worker perform a task, understand what they are doing, and then do it themselves without falling over. They built a two-part system to make this happen. First, they created a "translator" called Humanoid-PoseNet. Think of this as a super-smart camera brain that watches a video of a human worker. It doesn't just see a person; it calculates exactly where every joint is in 3D space. But here's the tricky part: a human's body is shaped differently than a robot's. So, this brain instantly rewrites the human's movements into a version that fits the robot's specific body shape, ensuring the robot doesn't try to twist its joints in impossible ways.

Once the robot knows what to move, it needs to figure out how to move without crashing. This is where the second part, Humanoid-ActionNet, comes in. This is the robot's internal coach, trained using a "teacher-student" method. The "teacher" is a super-powerful version of the robot that lives in a computer simulation and knows everything about physics (like how heavy a brick is or how slippery the ground is). The teacher learns the perfect way to move to stay balanced. Then, it teaches a "student" version of the robot. The student is the one that will actually go out into the real world, but it is not allowed to rely on physics secrets; it has to learn just by feeling its own joints and watching the target movements, just like a real robot would. This ensures that when the robot leaves the computer and steps onto a real construction site, it doesn't get confused by the difference between the simulation and reality.

The team tested this system with 30 different construction tasks, from carrying heavy pipes and stacking materials to signaling with flags and walking on uneven ground. They started by recording five human volunteers performing these tasks in a motion-capture studio. Then, they let their robot system learn from these recordings. The results were promising: the robot successfully learned to perform eight of these complex construction actions. In the computer simulations, the robot tracked the human movements with an average error of about 82.45 mm (Mean Per Joint Position Error) across its joints. When they put the robot to the test in the real world, it managed to carry pipes, hold concrete blocks, and even wave a flag while maintaining its balance, proving that the "translation" and "coaching" worked.

However, the paper is careful not to call this a perfect solution. The researchers noted that while the robot could do these tasks, it wasn't flawless. In the simulations, some attempts failed, with the robot losing its balance or stumbling, especially when the physics of the real world (like friction or weight) didn't match the computer model perfectly. They also pointed out that the robot's hands aren't quite as dexterous as a human's, so while it could hold a block, it couldn't necessarily grasp it with the same finesse as a human worker. The study suggests that this approach is a solid first step toward having robots that can work alongside us on construction sites, but there is still a lot of work to be done to make them as reliable and adaptable as the humans they are trying to mimic. The system showed that it is possible to teach a robot to move like a worker, but the journey from "learning to walk" to "building a skyscraper" is just beginning.

Technical Summary: Perception-and-Action System for Humanoid Robot Task Execution in Construction

Problem Statement

Construction sites are complex, hazardous, and labor-intensive environments where automation is increasingly necessary to address safety risks and labor shortages. While specialized robots (e.g., bricklaying masons, legged inspection bots) have been deployed, they are often single-task specific and lack the versatility to collaborate naturally with human workers in human-designed spaces. Humanoid robots offer a promising solution due to their human-like morphology, which aligns with existing tools and workflows, and their high degree of freedom (DoF), allowing for multi-task adaptability.

However, deploying humanoid robots in construction faces two primary technical gaps:

Morphological Mismatch: Existing human pose estimation methods output human-centered data that cannot be directly applied to humanoid robots due to differences in limb proportions, joint structures, and actuation constraints.
Physical Feasibility: Even if human motions are retargeted to a humanoid skeleton, converting these kinematic references into physically executable whole-body actions that maintain balance, contact stability, and dynamic feasibility in contact-rich construction environments remains a significant challenge.

Methodology

The authors propose a Vision-based Perception-and-Action (VPA) system designed to enable humanoid robots to learn construction tasks directly from worker demonstrations. The system consists of two deep learning modules: Humanoid-PoseNet and Humanoid-ActionNet.

1. Humanoid-PoseNet: Perception and Retargeting

This module bridges the gap between visual worker demonstrations and robot-executable poses. It comprises two sub-networks:

3D Human Pose Estimation: Utilizes off-the-shelf models (specifically PoseNet was selected as optimal) to reconstruct 3D human keypoints from 2D RGB video frames.
Human-to-Humanoid Retargeting: A deep network with a shared latent space architecture (two encoders and one decoder) maps the estimated human poses to humanoid-compatible configurations.
- Architecture: Uses Multi-Layer Perceptrons (MLPs) with a shared latent space.
- Training Objective: The network is trained using a combination of three loss functions to ensure geometric alignment and physical feasibility:
  - Triplet Loss ( $L_1$ ): Uses Bone-Direction Angular Error (BAE) to group similar human-humanoid pose pairs in the latent space.
  - Reconstruction Loss ( $L_2$ ): Ensures the decoder can accurately reconstruct robot poses from robot-encoded inputs.
  - Latent-Consistency Loss ( $L_3$ ): Enforces that human poses decoded to robot poses remain consistent when re-encoded, promoting cross-domain distribution alignment.

2. Humanoid-ActionNet: Whole-Body Control

This module learns a control policy to execute the retargeted poses while maintaining dynamic stability. It employs a Teacher-Student Reinforcement Learning (RL) framework:

Teacher Policy ( $\pi_{teacher}$ ): Trained in simulation (IsaacGym) using Proximal Policy Optimization (PPO). It has access to "privileged" observations (e.g., exact joint differences, root velocity) to maximize task rewards while enforcing physical constraints.
Student Policy ( $\pi_{student}$ ): Distilled from the teacher to operate without privileged observations. It relies only on proprioceptive states (joint positions/velocities) and a short history of past states, facilitating robust sim-to-real transfer.
Reward Design: The reward function is "physics-aware," incorporating terms for:
- DoF and keypoint position tracking.
- Root linear velocity and direction.
- Body rotation and upper-body coherence.
- Foot contact consistency (slip penalty) and air-time shaping.
Deployment: The student policy outputs target joint angles, which are executed on the physical robot via a Proportional-Derivative (PD) controller.

Experimental Setup

Data Collection: Five human subjects performed 30 construction-related actions (e.g., carrying pipes, lifting blocks, signaling, stair climbing) in a motion-capture environment.
Robot Platform: The Unitree G1 humanoid robot (23 actuated DoFs) was used for both simulation and physical deployment.
Evaluation Protocol:
1. Retargeting Accuracy: Measured via Mean Per Joint Position Error (MPJPE) on a held-out test set.
2. Sim-to-Sim Transfer: Evaluated across two physics engines (IsaacGym and MuJoCo) to test robustness against dynamics modeling differences.
3. Sim-to-Real Transfer: Deployed on the physical Unitree G1 to execute eight selected construction actions.

Key Results

Pose Retargeting: The Humanoid-PoseNet achieved an average MPJPE of 48.46 mm for the 14-joint humanoid representation, successfully translating diverse human postures into feasible robot configurations.
Task Execution: The Humanoid-ActionNet successfully executed eight construction-related actions (e.g., pushing a wheelbarrow, carrying a pipe, signaling) in both simulation and on the physical robot.
Tracking Performance: The system achieved an average motion-tracking error of 82.45 mm MPJPE across the eight tested actions.
Comparative Performance: In comparative studies against state-of-the-art whole-body control baselines (ExBody, OmniH2O, ExBody2), the proposed Humanoid-ActionNet demonstrated lower average MPJPE, indicating superior tracking accuracy for construction-specific motions.
Failure Analysis: The study acknowledged failure cases during sim-to-sim transfer (e.g., loss of balance, tracking breakdown), attributing them to difficulties in modeling contact-rich dynamics and simulation-to-hardware gaps.

Significance and Claims

The authors position this work as an early step toward deploying humanoid collaborators in construction. The paper claims the following contributions:

Novel Pipeline: It is the first method to enable a humanoid robot to learn and execute construction tasks directly from human demonstrations, addressing the specific challenges of the construction domain.
Bridging the Gap: The VPA system effectively bridges the morphological gap between humans and robots and the dynamic gap between kinematic imitation and physically stable execution.
Practical Applicability: By demonstrating successful sim-to-real transfer on a physical robot for tasks like material handling and signaling, the study provides a foundational step for implementing robotics in civil engineering.
Adaptability: The system is shown to be adaptable to different humanoid configurations (tested on both Unitree G1 and H1 kinematics), suggesting potential scalability to other humanoid platforms.

The authors remain modest regarding the current limitations, noting that the system currently focuses on posture-based imitation without explicit modeling of tool/material interaction, and that robustness in highly dynamic, contact-rich environments requires further refinement of reward functions and simulation realism.