Technical Summary: PanoVine – Whole-Body Visuomotor Control for Soft Growing Vine Robot

Problem Statement

Vine robots, a class of pneumatic soft growing continuum robots, offer unique advantages for navigating confined, complex, and unstructured environments (e.g., pipes, underground infrastructure) due to their compliant bodies and self-supporting growth mechanisms. However, controlling these robots for autonomous navigation and manipulation remains a significant challenge due to two primary factors:

1. Unpredictable Dynamics: The robot's behavior is governed by material compliance, hysteresis, buckling, body-environment interactions, and tension in the un-everted tail. These factors make the system difficult to model physically. Consequently, open-loop replays of action trajectories often result in drastically different robot configurations across trials.
2. Limited Onboard Sensing: The robot undergoes intricate whole-body deformations over large workspaces. Traditional limited sensing modalities (e.g., a single camera or IMU) are insufficient to capture the necessary state information for reliable closed-loop control, particularly when tip-mounted sensors add mechanical complexity or when the robot's body occludes its own view.

Existing model-based approaches rely on kinematic assumptions (e.g., piecewise constant curvature) or obstacle geometry priors that fail in cluttered, unknown environments. Similarly, prior learning-based methods have largely been restricted to simulation or short, non-growing soft arms, failing to scale to long, continuously growing vine robots with high degrees of freedom.

Methodology

The authors present PanoVine, the first autonomous vine robot system capable of closed-loop control using only onboard sensing. The approach integrates a custom hardware platform with a data-driven, vision-based control framework.

1. System Hardware: Distributed Whole-Body Vision

The PanoVine platform features a 6-meter long, 0.16-meter diameter soft growing body with seven controllable degrees of freedom (one extension, six revolute bending joints).

• Sensor Suite: The system is equipped with 19 RGB cameras distributed along the robot's body.
• Placement Strategy: Cameras are mounted on the TPU-coated side of the robot segments at a 60-degree angle relative to the body axis. They face either the distal or proximal ends. As the robot grows via eversion, cameras are gradually revealed, providing multi-perspective feedback of both the robot's deformation and the surrounding environment.
• Actuation: The robot is controlled via a joystick interface that commands joint-space steering and axial growth, with pressure adjustments for stiffness control.

2. Control Framework: Whole-Body Visuomotor Policy

The core of the system is an end-to-end visuomotor policy trained via imitation learning from teleoperated demonstrations.

• Input: The policy receives a history of observations $o = (I, q)$, where $I$ represents the RGB streams from all 19 cameras and $q$ represents proprioception (joint angles and base length).
• Architecture: The system utilizes a Diffusion Transformer backbone.
  • Vision Encoding: Images are resized to $224 \times 224 \times 3$ and processed through a pretrained CLIP Vision Transformer. The class token of each camera view serves as a compact embedding.
  • Cross-Attention: A cross-attention mechanism allows the policy to learn implicit multi-view correspondences and task-specific semantic relationships between the distributed visual inputs, proprioception, and actions, bypassing the need for explicit extrinsic calibration.
  • Action Output: The policy predicts an action chunk $a \in \mathbb{R}^{\tau \times d}$ (where $\tau=8$) specifying base length and joint angles for the next time steps.
• Key Design Choices for Robustness:
  1. Relative Representation: Proprioception and actions are expressed relative to the latest observation frame to ensure smooth transitions and robustness to hysteresis and actuation uncertainty.
  2. Data Rebalancing: To address the sparsity of steering actions in teleoperated data (which are dominated by pure growth), the training dataset is rebalanced to ensure the policy learns critical reactive steering behaviors (turning, climbing) rather than overfitting to growth.

Key Contributions

1. Hardware Platform: A vine robot system featuring distributed whole-body vision (19 cameras) that provides comprehensive, multi-perspective feedback of the robot state and environment throughout the growth process.
2. Control Framework: A learning-based visuomotor policy designed specifically for long, deformable robots. It effectively aggregates distributed sensing, maintains robustness to unreliable state estimation, and handles long-horizon behaviors with sparse steering actions.
3. Autonomous Demonstration: The first demonstration of a vine robot achieving autonomous closed-loop control in complex real-world scenarios without external sensing (e.g., GPS, motion capture) or model-based planning.

Experimental Results

The system was evaluated on two challenging real-world tasks:

1. Complex Course Navigation

• Task: Navigate a 6m course involving branch selection, 45° slope climbing, traversing unsupported gaps, obstacle avoidance, and sharp turns.
• Performance: The PanoVine policy achieved an 80% success rate.
• Baselines:
  • Trajectory Replay (Open-loop): 0% success rate. Failures included collisions with obstacles and failure to reach goals due to buckling-induced length discrepancies.
  • No Re-balancing: 10% success rate. The policy failed to steer sufficiently at branch points, highlighting the necessity of the data rebalancing strategy.

2. Object Reaching

• Task: Extend 2 meters and precisely steer to touch and knock down objects placed at unknown locations and orientations.
• Performance: The policy achieved an 85% success rate, demonstrating the ability to generalize across unseen objects and locations by grounding steering commands on multi-view visual feedback.
• Baselines:
  • Single Camera Policy: 0% success rate. The single camera failed to maintain view of the object during extension or steering, leading to missed targets.

Significance and Claims

The paper claims that PanoVine represents a significant step forward in soft robotics by demonstrating that autonomous closed-loop control of a vine robot is achievable using only onboard sensing.

The authors emphasize that their approach bypasses the limitations of traditional model-based pipelines, which struggle with the nonlinear, hysteretic, and discontinuous deformations inherent to soft growing robots. By learning an end-to-end visuomotor policy from distributed whole-body vision, the system can directly observe contact, deformation, and surrounding geometry, compensating for uncertainty in robot states and dynamics. This work establishes a foundation for future research into learning-based control for soft robots operating in complex, unstructured environments.