🤖 AI

HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environments

This paper presents HyReach, a real-time hybrid rigid-soft continuum manipulator system that integrates vision-based perception, 3D scene reconstruction, and learning-based control to achieve robust, generalizable object reaching in unseen cluttered environments with sub-2 cm accuracy.

Original authors: Shivani Kamtikar, Kendall Koe, Justin Wasserman, Samhita Marri, Benjamin Walt, Naveen Kumar Uppalapati, Girish Krishnan, Girish Chowdhary

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shivani Kamtikar, Kendall Koe, Justin Wasserman, Samhita Marri, Benjamin Walt, Naveen Kumar Uppalapati, Girish Krishnan, Girish Chowdhary

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to grab a specific apple from a tree, but the tree is surrounded by a dense, tangled thicket of branches, and you've never seen this particular tree before. If you were a rigid metal arm, you'd likely get stuck, snap a branch, or just fail to reach the fruit because you can't bend around the obstacles. If you were a purely soft, floppy noodle, you might be able to wiggle through, but you'd struggle to aim precisely or reach far enough.

HyReach is a new robotic system that solves this problem by combining the best of both worlds: the strength and reach of a rigid arm with the flexibility and "squishiness" of a soft arm. Think of it as a robotic octopus with a steel spine.

Here is a breakdown of how it works, using simple analogies:

1. The Robot: A "Steel Spine" with a "Soft Tentacle"

The robot is built in two parts:

The Base (The Steel Spine): This is a standard, rigid robotic arm (like the ones used in car factories). It provides the power and the long reach to get close to the target.
The Tip (The Soft Tentacle): Attached to the end of the rigid arm is a soft, bendable tube. This part can twist, curl, and squeeze through tiny gaps that a rigid arm couldn't fit through. It's like having a flexible snake tongue at the end of a crane.

2. The Eyes: "Exploring the Room"

The robot doesn't have a perfect map of the room. In fact, it might not even see the target at first because it's hidden behind a bush or a wall.

The Strategy: Instead of staring at one spot, the robot moves its head (a camera on its tip) around, taking pictures from different angles.
The Magic Brain: It uses a smart AI (called MAST3R) to stitch these pictures together into a 3D model of the room, almost like how your brain builds a 3D picture of a room when you walk around it. It also uses another AI (YOLO-World) to understand what it's looking at, even if it's never seen that specific object before (e.g., "Find the red box" or "Find the lemon").

3. The Planner: "The Safety-First Navigator"

Once the robot knows where the target is and what the obstacles look like, it needs to figure out how to get there. This is where the "Hybrid" part shines.

The Problem: If you just plan a path for a rigid arm, it might try to push through a branch and break it. If you plan for a soft arm, it might get tangled.
The Solution: The robot's brain plans a path that treats the two parts differently.
- The Rigid Part: Must never hit anything. It's like a car that must stay strictly on the road.
- The Soft Part: Is allowed to gently brush against obstacles. It's like a person squeezing through a crowded party; they might bump into shoulders, but they don't break anything.
Shape Awareness: The robot constantly calculates how its own soft body is bending. It doesn't just plan where the tip will go; it plans where the whole body will be to ensure it doesn't get stuck in a knot.

4. The Controller: "The Muscle Memory"

Finally, the robot has to actually move. Because the soft arm is so wiggly and unpredictable, traditional math is too slow and complicated to control it in real-time.

The Learning: The team trained a neural network (a type of AI) by letting the robot practice thousands of times. It's like teaching a dog to fetch by throwing a ball over and over until the dog knows exactly how to move its body to catch it.
The Result: When the robot sees a goal, it instantly knows exactly how to move its rigid joints and how much to squeeze its soft muscles to get there, without needing to stop and do complex math calculations.

Why Does This Matter?

The researchers tested this robot in four challenging scenarios:

Empty Room: Easy, but good for a baseline.
Obstacles: Blocks in the way.
Clutter: A messy bush where the target is hidden.
The Hole: A target on the other side of a wall with a tiny hole in it.

The Results:

Rigid Robots failed miserably in the clutter and hole scenarios because they couldn't bend.
Soft-Only Robots (or those without the smart planning) got tangled or couldn't aim precisely.
HyReach succeeded in almost all cases, reaching the target with an error of less than 2 centimeters (about the width of a thumb).

The Big Picture

This paper shows that by giving robots a "hybrid" body and teaching them to "see" and "feel" their way through messy, unknown environments, we can finally build machines that can work in the real world—like picking fruit in a messy orchard, searching through rubble after a disaster, or helping doctors navigate inside the human body—without needing a perfectly clean, pre-mapped factory floor.

It's the difference between trying to navigate a crowded dance floor with a rigid broom versus using a flexible, dancing partner who can squeeze through the crowd and grab your hand.

1. Problem Statement

Robotic manipulation in unstructured, cluttered, and previously unseen environments (e.g., agriculture, disaster relief) poses significant challenges.

Rigid Manipulators: While precise, they lack the dexterity and passive compliance to navigate tight spaces or interact safely with obstacles without complex, pre-calibrated modeling.
Soft Continuum Arms (SCAs): Highly adaptable and safe, but they suffer from limited workspace, difficult modeling of non-linear dynamics, and challenges in precise control.
The Gap: Existing hybrid systems (combining rigid and soft segments) often rely on perfect environmental knowledge, specific retraining for new scenes, or expensive sensors (like fiber optics) for shape sensing. There is a lack of a unified, real-time framework that enables open-world reaching (generalizing to unseen scenes) using only standard vision, while explicitly reasoning about the deformable geometry of the soft segment to ensure safety.

2. Methodology

The authors propose HyReach, a real-time, vision-guided framework consisting of three core modules:

A. Perception Module (Multi-View 3D Reconstruction)

Hardware: A 6-DoF rigid industrial arm (myCobot) equipped with a tri-chambered bending soft continuum arm (B3) at the distal end. A monocular RGB camera is mounted on the tip of the soft arm.
Process:
- The robot explores its immediate vicinity to capture multiple RGB viewpoints.
- Depth Estimation: Uses MAST3R (a state-of-the-art vision transformer) to estimate metric depth and reconstruct the 3D scene from monocular images.
- Goal Detection: Uses YOLO-World (an open-world object detector) to identify target objects via natural language queries (e.g., "red box," "lemon") without requiring fixed object categories.
- Scene Representation: The reconstructed point cloud is discretized into an occupancy grid to distinguish traversable space from obstacles.

B. Shape-Informed Path Planning

Algorithm: A modified RRT (Rapidly-exploring Random Tree Star)* algorithm.
Key Innovation: Unlike traditional planners that only check collisions at the end-effector, this planner evaluates collisions along the entire hybrid backbone.
- Rigid Segment: Enforces strict collision-free constraints ( $C_r = 0$ ).
- Soft Segment: Allows bounded, controlled contact ( $C_s \leq \tau$ ), treating compliance as a planning primitive rather than a disturbance.
Shape Estimation: Uses a Constant Curvature (CC) model to estimate the real-time shape of the soft arm based on actuator inputs. This shape is updated dynamically during tree expansion to ensure the planned path is kinematically valid and safe for the deformable geometry.
Collision Thresholding: A tunable hyperparameter ( $\tau$ ) defines the maximum allowable collisions for the soft segment, balancing safety and path feasibility.

C. Learning-Based Hybrid Controller

Architecture: A 20-layer Multi-Layer Perceptron (MLP) with a bottleneck architecture and ResNet blocks.
Input: Concatenation of the current pose and the relative pose to the goal.
Output: Control commands for both the rigid joint angles and the soft arm actuators.
Training: Trained on 9,536 data points collected by systematically exploring the 9-DoF workspace. The model minimizes Mean Squared Error (MSE) between predicted and ground-truth actuations.
Advantage: This model-free approach avoids the need for complex analytical modeling of the hybrid system's non-linear dynamics, enabling closed-loop control to arbitrary 6-DoF poses.

3. Key Contributions

First Open-World Hybrid Reaching System: Demonstrates the first hybrid manipulator capable of reaching objects in cluttered, unseen environments without environment-specific retraining or calibration. Achieves sub-2cm accuracy.
Lightweight Multi-View Perception: Develops a pipeline using only RGB cameras (no depth sensors) for 3D reconstruction and obstacle-aware planning, making it suitable for payload-limited robots.
Shape-Aware Planning: Proves that explicitly incorporating soft-arm shape estimation into the planning loop is critical for safety and success in cluttered environments, preventing entanglement and severe collisions.

4. Experimental Results

The system was tested in four increasingly difficult real-world environments: No Obstacles, Obstacles, Clutter (dense foliage), and Hole (narrow aperture).

Performance vs. Baselines:
- HyReach vs. Rigid Only: In the "Clutter" setup, HyReach achieved a 90.9% success rate (SR@2cm), whereas the rigid-only baseline dropped to 54.5% due to collisions.
- HyReach vs. Img2Act (Visual Servoing): The learning-based visual servoing baseline required retraining for each environment and failed in unseen scenes. HyReach generalized without retraining, achieving 75% success in "Obstacles" vs. the baseline's 54.5%.
Ablation Studies:
- Shape Estimation: Removing shape estimation reduced success rates significantly in cluttered environments (e.g., dropping from 90.9% to 54.5% in "Clutter") due to entanglement.
- Collision Thresholds: A moderate threshold ( $\tau = 4-6$ ) provided the best balance between safety and path feasibility.
- Controller Inputs: Providing both current pose and relative goal pose yielded the lowest translation error (1.2 cm) compared to other input configurations.
Accuracy: The system maintained translation errors below 2 cm in most environments and below 3 cm in the challenging "Hole" scenario.

5. Significance

This work bridges the gap between the adaptability of soft robotics and the precision of rigid robotics. By integrating vision-based 3D reconstruction, shape-aware planning, and learning-based control, HyReach enables robots to operate safely and effectively in complex, unstructured real-world scenarios where traditional rigid arms fail and pure soft arms struggle with precision and reach. The system's ability to generalize to unseen environments without retraining makes it a significant step toward deploying adaptive manipulators in fields like agriculture, search and rescue, and medical intervention.