💻 computer science

ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics

This paper introduces ManiSoft, a new benchmark and simulator designed to address the challenges of vision-language manipulation in soft continuum robotics by providing a scalable pipeline for generating diverse tasks and trajectories, while revealing current limitations in proprioceptive estimation and adaptive obstacle avoidance.

Original authors: Ziyu Wei, Luting Wang, Chen Gao, Li Wen, Si Liu

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyu Wei, Luting Wang, Chen Gao, Li Wen, Si Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot arm. Most of the ones you see in movies or factories are like human arms made of stiff metal bones and joints. They are great at moving in straight lines, but if you put a pile of books in their way, they often get stuck because they can't bend around the mess.

Now, imagine a different kind of robot arm: one made of soft, squishy material, like a giant, high-tech octopus tentacle. This is a soft continuum robot. It can twist, stretch, and squeeze its way around obstacles that would stop a stiff robot dead in its tracks.

The paper you're asking about, ManiSoft, is like a new "training gym" and "test track" designed specifically to teach these squishy arms how to follow human voice commands.

Here is the breakdown of what they did, using some everyday analogies:

1. The Problem: The "Stiff" vs. "Squishy" Dilemma

The Stiff Robot: Think of a rigid robot arm like a fishing rod. It's strong and precise, but if you need to reach around a tree to catch a fish, the rod can't bend. It just hits the tree.
The Soft Robot: Think of a soft robot arm like a garden hose or a snake. It can flow around the tree. However, controlling a snake is much harder than controlling a fishing rod. You can't just say "move the tip to point A" because the whole body bends in weird ways. You don't know exactly where the "joints" are because the arm doesn't have hard joints; it just squishes.

2. The Solution: ManiSoft (The Training Gym)

The researchers built a special video game (a simulator) to teach these soft arms how to listen to language commands like "Pick up the red cup" or "Stack the books."

The Simulator: They created a digital world where the soft arm behaves realistically. It's like a physics engine that knows how rubber stretches and bounces. They combined two types of physics: one for the squishy arm and one for the hard objects it touches, linking them together with a "virtual spring" so they move together naturally.
The Tasks: They set up four specific challenges, like a video game level:
1. Collecting: Grabbing an object and putting it in a box.
2. Aligning: Turning an object to face a specific direction.
3. Stacking: Piling items up neatly (like stacking pancakes).
4. Arranging: Placing items in a specific pattern on a table.

3. How They Made the Data (The "Teacher")

You can't just tell a soft robot what to do; it needs to see examples first. But writing down every single movement for a squishy arm is impossible for a human.

So, they built an automated teacher:

The High-Level Planner: This is like a project manager. It looks at the goal (e.g., "Stack the cups") and breaks it down into a list of checkpoints (e.g., "Move hand to cup," "Grab cup," "Move to stack"). It doesn't worry about how the arm bends, just where it needs to go.
The Low-Level Executor: This is like the muscle memory. It takes those checkpoints and figures out exactly how much pressure to apply to the soft arm to get there. It uses a "trial-and-error" learning method (Reinforcement Learning) to figure out the right amount of squeeze and twist.

They used this system to generate 6,300 different scenarios, ranging from empty tables to messy tables full of obstacles, creating a massive library of "expert" demonstrations.

4. The Results: The "Clean" vs. "Messy" Reality

They tested three different AI brains (policies) on this new gym to see if they could learn from the data.

The Good News: When the table was clean and simple (no obstacles), the AI models did a decent job. They could follow instructions and move the soft arm to grab things.
The Bad News: When they made the table messy (adding random obstacles, changing lighting, and using different words for the same objects), the AI got confused.
- The "Blindness" Issue: The AI struggled to guess where the arm actually was just by looking at a camera. Because the arm is squishy, it looks different from every angle. The AI couldn't tell if the arm was bent too far or not far enough.
- The "Squishy" Issue: The AI didn't know how to use the arm's flexibility to its advantage. Instead of bending around a block to grab a toy behind it, the AI often tried to push straight through the block and failed.

5. The "Stop-Moving" Glitch

One funny observation they made was that one of the AI models (OpenVLA) sometimes got stuck in a loop. After it successfully grabbed an object, it would just freeze and stop moving, even though the task wasn't finished. It was like a robot that got so nervous about holding the object that it forgot to put it down. The other models didn't have this problem as much.

Summary

ManiSoft is a new benchmark that says: "We have these amazing, flexible, octopus-like robots, but we don't have a good way to teach them to listen to us yet."

They built a simulator and a dataset to help researchers train these robots. Their tests showed that while current AI is getting better, it still struggles to "see" the shape of a squishy arm and figure out how to use that squishiness to navigate a messy world. It's a first step toward making robots that can work in our cluttered, real-world homes without getting stuck.

Technical Summary: ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics

Problem Statement
Existing vision-language manipulation (VLM) research predominantly targets rigid robotic arms, which rely on accurate proprioception and low-dimensional kinematics. While effective in structured environments, rigid morphologies face fundamental limitations in cluttered or confined spaces where significant shape adaptation is required. Soft robotic arms offer a promising alternative due to their continuous deformability, enabling them to navigate around obstacles. However, applying VLM to soft arms introduces distinct challenges: the lack of reliable proprioceptive sensors (often requiring inference from visual data alone), the complexity of distributed low-level actuation (e.g., torques, pressures), and the high-dimensional, coupled action space. Furthermore, existing simulators either excel at modeling elastic dynamics but lack robust contact interactions, or handle contacts well but fail to simulate continuous deformation.

Methodology
To address these gaps, the authors introduce ManiSoft, a benchmark designed to catalyze research in vision-language manipulation for soft arms. The methodology comprises three core components:

Hybrid Simulator (ManiSoft-Sim):
The authors develop a tailored simulator that bridges the gap between soft-body dynamics and rigid-body interactions.
- Soft Body Modeling: The arm is modeled as a Cosserat rod using the Elastica library, discretized into $N$ segments to capture axial, shear, bending, and torsional strains under external torques.
- Interaction Modeling: The end-effector (EEF) and environmental contacts are handled by MuJoCo for efficient contact and friction simulation.
- Coupling: A novel elastic force constraint links the deformable soft body and the rigid EEF. Relative translations ( $\Delta x$ ) and rotations ( $\Delta \theta$ ) between the attachment points generate restoring forces and torques via Hooke's law, ensuring coordinated yet compliant motion.
- Rendering: Visual observations are generated using Blender from fixed camera viewpoints.
Task Definition and Data Generation:
ManiSoft defines four manipulation tasks, each highlighting different aspects of deformable control:
- Collection (COLL): Gathering an object into a container.
- Alignment (ALN): Positioning an object to a specific 6-DoF pose.
- Stacking (STK): Assembling items in a stable vertical pile.
- Arrangement (ARR): Placing objects in a specified spatial configuration.
The benchmark includes 6,300 scene-trajectory pairs (2,100 clean, 4,200 randomized). An automated pipeline generates these scenes using an asset library of 263 objects. Crucially, the benchmark excludes internal proprioceptive states (e.g., joint angles) from the policy input, forcing models to infer the arm's configuration solely from visual observations.
Hierarchical Trajectory Generation:
To produce high-quality expert demonstrations at scale, the authors employ a hierarchical mechanism:
- High-Level Planning: A rule-based planner decomposes tasks into a sequence of 6-DoF waypoints.
- Low-Level Control: A Reinforcement Learning (RL) executor, trained with a dense reward function (combining pose difference and stability rewards), generates torque commands to drive the soft arm between waypoints. This decomposition mitigates the difficulty of directly planning torque sequences over long horizons.

Key Contributions

ManiSoft Benchmark: The first benchmark specifically tailored for vision-language manipulation with soft continuum robots, featuring a hybrid simulator, diverse tasks, and a large-scale dataset of expert trajectories.
Hybrid Simulation Framework: A novel integration of Cosserat rod dynamics and rigid-body contact simulation via elastic force constraints, enabling realistic contact-rich manipulation with deformable bodies.
Systematic Evaluation: A comprehensive benchmarking of three representative policy models (Diffusion Policy, RDT, and OpenVLA-OFT) under both clean and randomized conditions, revealing specific failure modes unique to soft embodiments.

Results
The authors evaluated three models: Diffusion Policy (DP), RDT-1B, and OpenVLA-OFT.

Clean Scenes: Existing models show promising results, with DP and OpenVLA-OFT achieving average success rates of ~31% and ~30%, respectively, significantly outperforming RDT (9.2%). DP excels in simpler tasks (COLL), while OpenVLA-OFT demonstrates stronger generalization in complex tasks (ALN, STK, ARR).
Randomized Scenes: All models experience a substantial performance drop. DP's success rate falls by 29.4%, whereas OpenVLA-OFT shows greater robustness, maintaining an average success rate of 27.0% (a drop of only 3.4% relative to its clean performance, and outperforming DP by 13.1% in the randomized setting).
Failure Analysis: Visualization of failure cases reveals three primary bottlenecks:
1. Proprioceptive State Ambiguity: Inaccurate visual estimation of the soft arm's deformation leads to uncorrected internal torques and unstable control.
2. Under-utilization of Compliance: Policies often fail to leverage the arm's deformability to navigate around obstacles, instead attempting direct paths that result in collisions.
3. Stop-Moving Behavior: Deterministic policies (like OpenVLA-OFT) occasionally enter a feedback loop where subtle visual changes after grasping suppress further action generation, a behavior less common in stochastic diffusion policies.

Significance and Claims
The paper positions ManiSoft as a valuable testbed for bridging the gap between rigid and soft arms in the context of vision-language manipulation. The authors claim that the benchmark exposes failure modes unique to deformable embodiments, specifically the challenges of visual proprioception and adaptive obstacle avoidance. They anticipate that ManiSoft will facilitate the development of methods that address these specific challenges, ultimately supporting the creation of safer, more human-friendly robotic systems capable of operating in complex, unstructured environments. The work is presented as an initial step, acknowledging limitations in actuation diversity and sim-to-real transfer, but establishing a foundation for future research in compliant interaction.