🤖 machine learning

Learning to build covering structures with continuous adjustments

This paper presents HSAC, a reinforcement learning framework that enables robots to adaptively construct complex structures by generating sequences in real-time without predefined plans, utilizing graph neural networks and an extended Soft Actor-Critic algorithm to handle hybrid discrete-continuous action spaces and successfully transferring from simulation to physical hardware.

Original authors: Gabriel Vallat, Maryam Kamgarpour, Stefana Parascho

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gabriel Vallat, Maryam Kamgarpour, Stefana Parascho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots could build complex structures not by following a rigid, pre-drawn blueprint, but by feeling their way through the process, adjusting to every tiny wobble and imperfection as they go. This is the promise of robotic construction, a field that aims to use materials more efficiently and create shapes that are too difficult for human hands or traditional machines to achieve. However, a major hurdle has always stood in the way: the real world is messy. No matter how precise a plan is, physical materials have slight variations, and machines make small errors. In traditional construction, if a block is placed even a fraction of a millimeter off, the entire plan might need to be scrapped and restarted, or worse, the structure could collapse. Current methods struggle to adapt to these inevitable mistakes because they rely on fixed instructions that cannot account for the unpredictable nature of physical reality.

A team of researchers has developed a new way for robots to learn how to build, one that abandons the idea of a master plan entirely. Instead of following a script, their robots learn to build by trial and error, using a type of artificial intelligence known as reinforcement learning. In this approach, the robot acts like a student who learns by doing: it tries to place a block, sees what happens, and if the structure remains stable, it gets a reward; if it wobbles or falls, it learns from the mistake. The researchers focused on a specific challenge: building a spanning arch using two robots working together. One robot places the blocks, while the other holds the free end to keep the structure from tipping over. This setup is delicate; the structure is always on the edge of instability, requiring constant, subtle adjustments to stay upright.

The core of their discovery is a new algorithm that allows the robot to make two types of decisions at once: choosing which type of block to use and deciding exactly where to place it. This is a difficult problem because the choice of block and the placement position are deeply connected; the best spot for one type of block might be terrible for another. Previous methods tried to handle this by simplifying the problem or sticking to rigid plans, but the researchers found that these approaches often got stuck in local solutions, unable to find the best way to build. Their new method, which they call a hybrid soft actor-critic algorithm, treats the construction process as a continuous conversation between the robot and the structure. It allows the robot to explore different possibilities, learning not just what works, but how to recover when things go slightly wrong.

To make this work, the researchers represented the structure not as a list of coordinates, but as a network of connections, similar to how a map shows how cities are linked by roads. This graph-based view helps the robot understand the shape of the building regardless of how it is rotated or shifted in space. A key innovation in their system was a specific way of organizing this information to ensure that the robot's decisions about which block to pick did not get confused by its decisions about where to put it. By structuring the information carefully, the robot could learn to explore a vast number of possibilities without getting overwhelmed, eventually finding a path to a stable arch that other methods missed.

The team tested their system in a simulated environment first, where they could run thousands of trials quickly. They compared their new algorithm against an existing method and found that their approach was significantly better at learning. While the older method often got stuck in a loop of suboptimal solutions, never quite reaching the perfect arch, their new system consistently found better ways to build. It was also robust, meaning it worked well even when the researchers changed the settings or made the task harder by adding more types of blocks to choose from. The system could handle up to ten different block types without losing its ability to learn, showing that it could scale to more complex tasks.

To prove that this learning could translate to the real world, the researchers built a physical setup using two industrial robots and a set of 3D-printed plastic blocks. They placed a camera above the workspace to track the position of every block as it was placed. The robot used the camera's data to update its understanding of the structure and decided where to place the next block based on what it had learned in the simulation. The result was a successful arch built in real time, with the robots adjusting their actions as the structure grew. When they compared this real-world, adaptive building process to a traditional method where the entire plan was calculated beforehand, the adaptive approach performed much better. The structure built by the learning robot stayed true to its intended shape, while the pre-planned version drifted off course due to small, unavoidable errors.

This work demonstrates that robots can learn to build complex structures by adapting to the physical world in real time, rather than trying to force the world to fit a perfect plan. By letting the robot learn from its own mistakes and successes, the researchers have shown a path toward construction that is more resilient, efficient, and capable of handling the messy reality of the physical world. The success of their physical experiment suggests that this approach could eventually be used to build with a wide variety of materials, from plastic to stone, creating structures that are both beautiful and strong, built by machines that understand the subtle art of balance.

Technical Summary: Learning to Build Covering Structures with Continuous Adjustments

Problem Statement
Robotic construction offers potential for material efficiency and complex geometry creation but is currently constrained by rigid, high-precision pre-planning. Physical fabrication inherently involves tolerances, inaccuracies, and unexpected changes that cause mismatches with idealized plans, potentially leading to catastrophic failure or costly revisions. Existing plan-free methods often rely on strong simplifications, such as discretizing space or limiting scope to 2D structures, or they struggle with the computational cost of physics-based stability checks in complex environments. Furthermore, current reinforcement learning (RL) approaches struggle with the specific challenge of mixed action spaces required in robotic construction, where an agent must simultaneously select a discrete block type and determine continuous placement parameters. These decisions are coupled, and suboptimal choices can lead to irrecoverable structural collapse.

Methodology
The authors propose a model-free reinforcement learning framework that treats construction as a controlled dynamical system, generating construction sequences adaptively at each time step rather than following a predefined plan.

State Representation: The structure is modeled as a labeled heterogeneous graph ( $G_s$ ). Nodes represent current blocks ( $N_b$ ), possible block types to add ( $N_a$ ), and the objective area ( $N_o$ ). Edges link these nodes. Crucially, the method attaches a reference frame to each block and labels edges with relative positions rather than using a global coordinate system. This design ensures the state representation is invariant to rotation, translation, and permutation, allowing the agent to learn more efficiently.
Action Space: The action space is hybrid. The agent first selects a discrete block type ( $a \in \{1, \dots, N\}$ ) and then determines a continuous placement parameter ( $u \in \mathbb{R}^4$ ). The continuous parameters define a 2D translation and an angle (represented by sine and cosine) to orient the new block relative to the previous one.
Algorithm (HSAC): The authors introduce Hybrid Soft Actor-Critic (HSAC), an extension of the Soft Actor-Critic (SAC) algorithm to mixed action spaces.
- Policy Decomposition: The policy is factorized into a discrete component ( $\pi^d_\phi(a|s)$ ) and a continuous component ( $\pi^c_\phi(u|a,s)$ ). The algorithm optimizes these separately using distinct entropy coefficients ( $\alpha_d$ and $\alpha_c$ ) to prevent the discrete policy from collapsing to deterministic behavior, a common failure mode in mixed-action settings.
- Graph Neural Networks (GNNs): The state is processed using GNNs (specifically Transformer convolutional networks) to extract features.
- Unidirectional Edges for Q-Function: A key technical novelty is the use of unidirectional edges in the input graph connecting state nodes to action nodes. This structural inductive bias enforces independence between the Q-values of different discrete actions for the same continuous parameters, solving the over-parametrization issues found in standard extensions of Deep Q-Networks (DQN) to mixed spaces.
Simulation Environment: The environment uses Rigid Block Analysis (RBA) for stability checks. The transition function simulates placing a block, releasing support, and checking for collapse. Rewards are proportional to the covered area, with a terminal reward for successfully spanning the target area.

Key Contributions

HSAC Algorithm: The first sample-efficient learning algorithm for mixed continuous-discrete action spaces in the context of costly physics simulations. It extends SAC by splitting the policy cost and utilizing separate entropy coefficients.
Graph-Based Inductive Bias: The introduction of a graph-structured state representation with relative reference frames and unidirectional edges to the action space. This ensures invariance to geometric transformations and structurally enforces Q-value independence, which is critical for stable learning.
Real-World Validation: Successful deployment on a physical two-robot setup (ABB GoFa arms) using 3D-printed blocks, demonstrating that policies trained in simulation transfer to real hardware in a closed-loop execution.

Results

Performance vs. Baselines: HSAC demonstrates significantly higher asymptotic performance compared to the prior state-of-the-art hybrid-PPO (HPPO). While HSAC steps take longer to compute, the algorithm converges to better policies and avoids getting stuck in local minima where HPPO fails.
Robustness and Scalability: The algorithm shows robustness to hyperparameter choices and scales effectively to larger action spaces, handling up to 10 discrete block types without performance degradation. It also adapts to varied environmental dynamics, such as changes in friction coefficients.
Adaptability: In perturbation experiments, the policy successfully compensated for forced deviations in block placement during construction, resulting in structures nearly identical to the unperturbed baseline.
Physical Execution: In the physical prototype, the closed-loop HSAC system successfully built a spanning arch. Compared to an open-loop (pre-planned) baseline, the closed-loop system maintained better alignment with the target plane, confirming its ability to handle real-world inaccuracies.

Significance
The paper claims that this work represents a significant step toward plan-free robotic construction. By moving away from rigid pre-planning and utilizing adaptive, model-free RL, the method accommodates the inherent uncertainties of physical fabrication. The combination of graph-structured states, unidirectional edge architectures for mixed-action Q-functions, and the HSAC algorithm provides a generalizable framework for robotics problems that couple discrete and continuous decisions. The successful transfer from simulation to a physical two-robot system validates the practical viability of this approach for building complex, stable structures like arches using dry-jointed blocks.

Technical Summary: Learning to Build Covering Structures with Continuous Adjustments

More like this