💻 computer science

Module Number Adaptive Visual Shape Control for Serial Modular Soft Robots

This paper introduces a module number adaptive visual shape control method for serial modular soft pneumatic robots that enables scalable whole-body configuration control across varying numbers of modules and environmental conditions by reusing a single-module-trained controller through local image patch decomposition and geometric data augmentation.

Original authors: Kyohei Akamine, Takato Horii, Yusuke Sakaue, Hiroki Ishizuka

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kyohei Akamine, Takato Horii, Yusuke Sakaue, Hiroki Ishizuka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Soft robots are a distinct branch of engineering that replaces rigid metal and plastic with flexible, rubber-like materials. Unlike traditional machines that move with precise, fixed joints, these robots bend and twist like living tissue, allowing them to squeeze through tight spaces, handle delicate objects, or navigate uneven terrain without causing damage. Because their bodies are made of soft, squishy components, they do not have a single fixed shape; instead, their entire form can change continuously. This flexibility makes them incredibly useful for tasks in confined environments, such as medical procedures or search-and-rescue missions, but it also makes them notoriously difficult to control. Traditional computer models struggle to predict exactly how a soft robot will move because the material bends in complex, unpredictable ways. To solve this, researchers have turned to vision-based control, where a camera watches the robot and adjusts its movements in real time, much like a human watching their own hand to guide it. However, a major hurdle has remained: most of these visual control systems are built for a specific robot size. If a robot is made of five connected segments, the control software usually cannot be used on a robot made of three or seven segments without being completely rewritten and retrained from scratch.

A team of researchers has developed a new method that allows a single control system to manage soft robots of varying lengths without needing new training data for each size. They focused on a type of robot built by linking identical, inflatable segments together in a line. In their experiments, they constructed robots ranging from just one segment up to five segments connected end-to-end. The core of their discovery is a technique that breaks the robot's entire body image into small, individual pieces, treating each segment as a separate entity. Instead of trying to learn how a five-segment robot moves as one giant, complicated object, the system learns how a single segment moves and then applies that same knowledge to every segment it sees, regardless of how many neighbors it has. This approach means that once the researchers taught the computer how to control one single segment, that same "brain" could be instantly reused to control a robot with one, three, or five segments.

To make this work, the researchers had to overcome the fact that a segment looks different depending on where it sits in the line. A segment at the bottom of a five-segment robot appears in a different part of the camera's view and at a different angle than a single segment sitting alone on a table. To teach the computer to ignore these visual differences, they used a technique called geometric data augmentation during the training phase. This involved artificially rotating and shifting the images of the single segment in thousands of different ways, effectively showing the computer that the same piece of rubber could appear in many positions and orientations. By training the system on these varied views, the computer learned to recognize the essential shape of the segment rather than its specific location in the picture. When the system was deployed on a multi-segment robot, it used a specialized tool to locate each segment in the camera feed, crop it out, and feed it to the same control module. The module then calculated the necessary air pressure adjustments for that specific segment to match a desired target shape.

The researchers tested this system on physical robots suspended in a laboratory, using a standard webcam to watch them. They programmed the robots to bend into specific shapes, such as twisting or curving forward, and measured how closely the actual robot matched the target image. The results showed that the system successfully guided robots with one to five segments toward their target shapes. While the robots with more segments took slightly longer to settle and showed slightly more error than the shorter ones, the control system worked effectively across all configurations without any additional training. The system also proved robust against changes in the environment; it maintained control even when the lighting in the room changed, when objects were placed in the background, or when a heavy weight was attached to the tip of the robot. In one test, a 500-gram mass was attached to the end of a five-segment robot, simulating the extra load a real-world robot might carry, and the system still managed to guide the robot to the correct posture.

A critical part of the system's reliability was its ability to handle missing information. In a real-world scenario, parts of a robot might be hidden by its own body, by a person's hand, or by other objects, causing the camera to lose track of a specific muscle or segment. The researchers built a lightweight network that could guess the missing visual data based on the surrounding parts. When a piece of the robot's shape was obscured, this network reconstructed the missing image, allowing the control system to continue operating smoothly. Without this reconstruction, the robot would have struggled to move or failed to reach its target. The study demonstrated that by decomposing a complex, whole-body problem into simple, local tasks, a single control model can scale up to handle much larger and more complex machines. This finding suggests that future soft robots could be assembled and reassembled with different numbers of parts, and the same control software would work immediately, eliminating the need to collect new data or retrain the system every time the robot's configuration changes.

Technical Summary: Module-Number-Adaptive Visual Shape Control for Serial Modular Soft Robots

1. Problem Statement

Soft robots, particularly those constructed from modular pneumatic actuators, offer significant potential for operation in confined and unstructured environments. However, controlling the whole-body shape of these robots is challenging due to their high degrees of freedom, material nonlinearities, hysteresis, and manufacturing variabilities. While data-driven control methods have emerged to bypass complex analytical modeling, existing approaches typically rely on training controllers for fixed robot structures.

A critical limitation in current image-based whole-body shape control is the lack of scalability. When the number of modules in a serial soft robot changes, the spatial distribution of modules within the camera's field of view shifts. Consequently, a controller trained on a specific configuration (e.g., a single module) cannot be directly reused for configurations with different module counts without collecting new training data and retraining the model. This paper addresses the need for a control framework that can adapt to varying numbers of modules using control knowledge learned exclusively from a single-module configuration.

2. Methodology

The proposed framework achieves scalable whole-body control by decomposing the global control problem into local module-level tasks. The system consists of three primary components: a Module Extractor, a Module Controller, and a Mask Reconstruction Network.

A. System Architecture

Module Extractor:
- Utilizes a Module Segmenter (MS), an instance-segmentation model (based on Detectron2), to identify bounding boxes for each module in the whole-body image.
- The extracted modules are sorted by their vertical position (top to bottom) to establish a consistent correspondence between current and target images.
- The whole-body image is cropped into local patches corresponding to each module. An interpolation process is applied to the patch boundaries to reduce background noise.
- A single MS is fine-tuned on a combined dataset containing images from all tested configurations (1 to 5 modules), allowing it to localize modules regardless of the total count.
Module Controller:
- Operates on a closed-loop basis, iteratively updating actuation voltages based on the difference between the current and target states.
- Input Processing: For each module patch, an Actuator Segmenter (AS) identifies the three pneumatic artificial muscles (left, center, right). The masks are reordered based on their centroid x-coordinates to ensure a consistent spatial channel order (Left-Center-Right).
- State Representation: The input tensor for the controller is a 9-channel concatenation of:
  - Current binary masks ( $M_{t,n}$ ).
  - Target binary masks ( $M_{targ,n}$ ).
  - A normalized voltage map ( $Q_{t,n}$ ) broadcast from the current command voltages.
- Prediction Model: A CNN-based voltage prediction model ( $I_\theta$ ) estimates the desired voltage vector. The final command is updated using a first-order gain ( $K=0.5$ ) applied to the difference between the predicted and current voltages, followed by clipping to the valid voltage range.
- Training Strategy: The controller is trained only on single-module visual and actuation data. To facilitate transfer to downstream modules in longer chains, geometric data augmentation (random rotations and translations) is applied during training. This simulates the positional and orientational variations that downstream modules would exhibit in a multi-module setup.
Mask Reconstruction Network:
- Addresses the issue of missing visual information caused by occlusion, overlap, or lighting changes.
- A lightweight U-Net-based network ( $R_\phi$ ) reconstructs missing actuator-mask channels.
- Mechanism: If a specific channel (e.g., the left mask) is missing or unreliable, the network takes the remaining two mask channels and the current voltage vector as input to predict the missing channel. This ensures the controller receives a consistent 9-channel input tensor even under partial occlusion.

3. Key Contributions

The paper outlines three primary contributions:

Compositional Visual Control Framework: A method that reuses a controller trained exclusively on single-module data for robots with varying numbers of modules (1 to 5) without requiring configuration-specific control data collection or controller retraining.
Scalable Transfer via Augmentation: Demonstration that geometric data augmentation (rotation and translation) applied during single-module training effectively reduces the visual distribution shift, enabling the controller to generalize to downstream modules in longer serial chains.
Robustness Validation: Experimental verification of the controller's performance on physical robots under varying module counts, environmental disturbances (lighting, background, affine transformations), payload loading, and synthetic mask loss.

4. Experimental Results

Experiments were conducted on a physical modular soft pneumatic robot (up to 5 modules, total length 875 mm) using a single RGB camera.

Scalability (1 to 5 Modules): The controller, trained solely on single-module data, successfully drove robots with 1 to 5 modules to converge toward target postures (twisted, forward, backward, etc.). While the mean squared error (MSE) and its standard deviation increased slightly with the number of modules (indicating error accumulation), the image error consistently decreased over time, converging within approximately 20 seconds.
Impact of Data Augmentation: Experiments comparing training with and without geometric augmentation showed that augmentation significantly suppressed the increase in error associated with increasing module counts, confirming its role in generalizing the control policy.
Environmental Robustness: The system maintained performance under altered illumination, background obstacles, and synthetic affine transformations (translation and rotation) of the feedback image.
Payload Loading: When a 500g payload was attached to the tip of the 5-module robot, the controller successfully converged to the target shape, although the variance in error across different postures increased.
Mask Reconstruction: In experiments where one actuator mask channel was synthetically removed, the reconstruction network successfully restored the missing visual features, allowing the robot to converge to the target posture. Without reconstruction, the robot failed to move effectively.

5. Significance and Limitations

The paper claims that this work enables scalable, markerless whole-body shape control for serial modular soft robots. The core significance lies in the ability to deploy a control model trained on a minimal, single-module dataset to complex, variable-length structures without retraining. This reduces the data collection burden and facilitates the reconfiguration of modular robots in dynamic environments.

The authors modestly acknowledge limitations:

Error Accumulation: Steady-state errors increase with the number of modules, attributed to changes in load distribution (gravitational moments) and the loss of depth information inherent in monocular vision.
Data Augmentation Limits: The current 2D rotation and translation augmentation does not fully reproduce apparent scale changes caused by depth motion (3D variations).
Target Generation: The study assumes a physically feasible target image is provided; it does not address the generation of such targets from high-level planners or sketches.

The results suggest that while single-module learning can be composed into whole-body control, the physical coupling and depth ambiguities in longer chains impose practical limits on precision, which future work may address through load compensation or 3D-aware augmentation.