Technical Summary: SWIM – Vision-Language-Grounded Soft Whole-Body Interactive Manipulation

1. Problem Statement

The paper addresses the challenge of vision-language-grounded whole-body manipulation for soft and continuum robots. While soft robots offer unique capabilities through distributed body deformation and contact (e.g., wrapping, enclosing), translating high-level language instructions and visual context into executable whole-body actuation sequences remains difficult.

The authors identify three fundamental mismatches that hinder existing approaches:

1. Action-Interface Mismatch: Standard Vision-Language-Action (VLA) models often output end-effector motions (Cartesian poses), which fail to specify the coordinated body deformation and distributed contact required for soft manipulation.
2. Whole-Body State Mismatch: End-effector states do not capture high-dimensional body configurations. Furthermore, action targets provide only indirect geometric supervision, making it difficult to learn the relationship between body deformation and actuation from limited demonstrations.
3. Execution Mismatch: Sim-to-real discrepancies and inference/communication latencies in online deployment cause commands to be applied to mismatched states, leading to failure in physical execution.

The objective is to map an initial RGB observation ($I$) and a language instruction ($\ell$) to a complete actuation-command sequence ($U$) that completes the task through whole-body deformation and distributed contact.

2. Methodology: The SWIM Framework

The proposed framework, SWIM, operates in two distinct stages: training and inference/deployment. It utilizes a tendon-driven spiral continuum robot (SpiRob) as the testbed.

A. SWIM-VLA: Learning with Geometric Supervision

The core policy, SWIM-VLA, is trained on simulated interaction trajectories. It builds upon the OpenVLA-OFT backbone and introduces two key components to address the state and action mismatches:

1. Shared Multimodal Representation: The policy encodes the RGB observation, language instruction, and the current tendon-length vector (providing low-dimensional proprioception) into a shared representation ($h_t$).
2. Diffusion Action Head: To address the multi-modal nature of soft robot control (where multiple actuation sequences can achieve the same task), the policy uses a diffusion model to predict the conditional distribution of $H$-step actuation command chunks. This allows the model to capture diverse valid strategies rather than a single deterministic output.
3. Visual Soft Proprioception (VSP): To address the lack of direct geometric supervision, an auxiliary VSP head is trained alongside the diffusion head. VSP predicts the image-plane coordinates of ordered "body anchors" distributed along the robot's backbone. By minimizing the localization error of these anchors against simulation ground truth, the shared representation is forced to retain whole-body geometric information, improving learning from limited demonstrations.

The joint training objective combines the diffusion loss ($L_{act}$) and the VSP loss ($L_{VSP}$). After training, the VSP head is discarded, and the encoder and diffusion head form the deployable policy.

B. Virtual Rollout and Embodied Mechanical Intelligence

During inference, SWIM avoids online policy queries on the physical robot to mitigate latency and sim-to-real gaps. Instead, it employs a virtual rollout strategy:

1. Initialization: The initial physical RGB observation is used to instantiate a virtual scene in a simulator (MuJoCo) with calibrated object models.
2. Iterative Virtual Rollout: The trained SWIM-VLA policy queries the simulated environment iteratively. It predicts command chunks, executes the first $K$ steps in simulation, updates the simulated state, and repeats until the task is complete.
3. Open-Loop Execution: The resulting complete sequence of commands is executed open-loop on the physical robot.
4. Role of Compliance: The framework relies on embodied mechanical intelligence. The robot's intrinsic compliance allows it to adapt locally to contact variations during physical execution without requiring real-time visual feedback or policy replanning.

3. Key Contributions

The paper claims three primary contributions:

1. SWIM Framework: A deployment strategy that generates complete actuation sequences via virtual rollouts, leveraging intrinsic body compliance for local contact adaptation and removing the need for online VLA inference during physical execution.
2. SWIM-VLA Policy: A tendon-state-conditioned policy that combines a diffusion action head with Visual Soft Proprioception (VSP). This design models conditional action distributions and uses geometric supervision to retain body shape information, enabling learning from limited demonstrations.
3. Comprehensive Evaluation: An evaluation on packing, reaching, and grasping tasks on a planar tendon-driven robot, demonstrating the efficacy of the virtual-rollout deployment compared to direct online deployment.

4. Experimental Results

The framework was evaluated on a planar SpiRob with 24 joints and two tendon actuators.

Simulation Results

On held-out test conditions, SWIM-VLA achieved the following success rates:

• Packing: 100%
• Reaching: 96%
• Grasping: 88%

These results outperformed an adapted OpenVLA-OFT baseline (90%, 72%, 52%) and ablated variants. Specifically:

• Removing VSP reduced reaching and grasping success to 90% and 78%, respectively, highlighting the importance of geometric supervision for distal placement.
• Replacing the diffusion head with a deterministic regressor reduced grasping success to 69%, confirming the value of modeling action distributions for contact-rich tasks.

Physical Deployment Results

The authors compared SWIM (virtual rollout + open-loop execution) against Direct SWIM-VLA (online queries on the physical robot) using the same policy checkpoint:

• Packing: SWIM achieved 100% (20/20) vs. 75% (15/20) for direct deployment.
• Reaching: SWIM achieved 80% (16/20) vs. 40% (8/20) for direct deployment.
• Grasping: SWIM achieved 75% (15/20) vs. 25% (5/20) for direct deployment.

While SWIM recovered a substantial portion of the performance lost in direct sim-to-real transfer, physical success rates remained slightly below simulation results, attributed to object-position errors during scene initialization and material/structural differences.

5. Significance and Claims

The paper positions SWIM as a solution to the specific challenges of translating language and vision into soft robot manipulation. Its significance lies in:

• Bridging the Sim-to-Real Gap: By shifting the policy interaction loop to a virtual environment and relying on the robot's physical compliance for the final execution, SWIM mitigates the issues of inference latency and model inaccuracies that plague online deployment.
• Geometric Awareness in VLAs: The introduction of VSP demonstrates that explicitly supervising body geometry improves the learning of whole-body actuation strategies from limited data, addressing the "whole-body state mismatch."
• Handling Multi-Modal Actions: The use of diffusion models acknowledges that soft robots often have multiple valid actuation paths for a single task, a nuance often lost in deterministic regression approaches.

The authors note that the current study is limited to planar manipulation, predefined reset configurations, and anchored grasping targets, with future work planned for 3D interactions and moving objects.