⚡ electrical engineering

From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation

This paper presents an open-vocabulary shared autonomy framework for robotic manipulation that combines wearable-free gesture control, vision-language target grounding, and GPU-accelerated model-predictive control to assist operators in achieving precise, collision-free grasps in cluttered industrial environments.

Original authors: Murilo Vinicius da Silva, Ricardo V. Godoy, Juliano Negri, Gustavo J. G. Lahr, Ranulfo Bezerra, Marcelo Becker

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Murilo Vinicius da Silva, Ricardo V. Godoy, Juliano Negri, Gustavo J. G. Lahr, Ranulfo Bezerra, Marcelo Becker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to thread a needle while wearing thick winter gloves, standing on a wobbly ladder, and looking at the needle through a foggy window. That is roughly what it feels like to control a giant robot arm from a distance. This is the world of teleoperation, where a human operator guides a machine using cameras and joysticks. The problem is that our eyes and brains aren't great at judging depth through a screen, and industrial environments are often cluttered with pipes, walls, and other obstacles that are easy to crash into.

To fix this, scientists have developed shared autonomy. Think of this as a "co-pilot" for the robot. Instead of the human doing everything alone, the robot's computer chips in to help. It might gently steer the arm away from a wall or nudge it toward a target, but the human still holds the steering wheel. This paper dives into a new, smarter version of this co-pilot system, designed specifically for robots that can walk around (like four-legged dogs) and do delicate tasks in messy, real-world factories.

The "Smart Co-Pilot" for Robot Dogs

Meet the robot: a four-legged machine (like a Boston Dynamics Spot) with a long, dexterous arm attached to its back. Its job? To walk into a messy industrial site, find a specific valve or tool, and turn it or pick it up. The challenge? The operator is far away, looking at the robot through a camera feed that makes depth perception tricky. If the operator tries to turn a valve, they might accidentally smash the robot's arm into a nearby pipe because they can't see exactly how close they are.

This paper introduces a system that acts like a helpful, invisible guide for the robot's arm. It combines three superpowers: seeing without markers, understanding language, and gentle steering.

1. No Gloves, No Calibration, Just You

Usually, to control a robot with your body, you need to wear special suits with sensors or stand in front of a camera that needs to be calibrated to your height. This new system says, "No thanks." It uses a standard 3D camera (RGB-D) to watch the operator's natural movements.

Imagine the camera is a super-observant friend who watches your shoulders, hips, and wrists. It builds a mental map of your body in real-time. If you move your right hand forward, the robot's arm moves forward. If you twist your wrist, the robot's gripper twists. The system even figures out your arm length on the fly, so it knows exactly how much to scale your movements to match the robot's size. No setup, no markers, just you and the robot moving in sync.

2. "Hey Robot, Grab That Wheel Valve"

In the past, telling a robot what to grab was hard. You had to point at a specific object or use a code. This system lets you use natural language. The operator can simply say (or type), "Wheel valve."

Here's where the magic happens: The robot uses a "vision-language model" (think of it as a robot brain that can read and see at the same time) to look at the camera feed and find the "wheel valve." Once it spots it, it doesn't just say "I see it." It creates a precise 3D target point in space, like a digital bullseye. Even better, it keeps tracking that valve as the robot moves around, using other cameras on its body to make sure it never loses sight of the target, even if the operator's view gets blocked.

3. The Invisible Magnetic Field

This is the paper's coolest trick. As the robot arm gets close to the target, an invisible "potential field" (like a gentle magnetic force) kicks in.

Imagine you are driving a car toward a parking spot. You are steering, but there's a gentle wind pushing your car slightly toward the center of the spot to help you park perfectly. That's what this system does.

You are still in charge: If you want to move left, the robot moves left. The system never takes the wheel away from you.
The help is subtle: As you get within 0.4 meters (about 1.3 feet) of the target, the system gently pulls your command toward the exact center of the "wheel valve." It corrects tiny errors you might make because of the foggy camera view.
The result: You can aim roughly, and the robot's "co-pilot" smooths out the final approach, ensuring the gripper lands exactly where it needs to be.

4. The "Don't Crash" Shield

While the robot is moving, it is constantly building a 3D map of the room using its own cameras. It knows exactly where the walls, pipes, and traffic cones are. The paper tested this by having the operator deliberately try to drive the robot arm into a traffic cone.

The result? The operator's command went straight into the cone, but the robot's arm stopped 18 cm (about 7 inches) away. The system acted like a forcefield, bending the path around the obstacle while still trying to follow the operator's general direction. It proved that even if the human makes a mistake, the robot's safety brain prevents a crash.

5. The "Auto-Pilot" Button

Sometimes, the operator is tired of steering. Once they have found the target and confirmed it, they can raise two fingers to switch to autonomous mode. The robot then takes over completely, using the same safety rules and the same 3D target to finish the job (like grabbing the valve) on its own. If the operator wants to take back control, they just raise one finger again.

What Did They Find?

The researchers tested this system on a real robot in a messy industrial setting with two main tasks: turning a large industrial valve and picking up a power drill.

The "Full Team" Wins: When they used the whole system (the language target + the gentle steering + the crash shield), the robot succeeded in 100% of the trials (5 out of 5 for both tasks).
The "Half-Team" Fails: When they removed the crash shield, the robot hit obstacles. When they removed the gentle steering, the robot missed the target because the camera view wasn't precise enough. This showed that both parts are necessary; they fix different problems.
The Numbers: The system was very accurate. The robot's arm followed the human's hand with an average error of only 59 mm (about 2.3 inches). In the crash tests, the operator tried to push the arm 6 cm into an obstacle, but the robot kept the arm at least 18 cm away from the danger.

The Bottom Line

This paper doesn't claim to have solved every problem in robotics. It shows that by combining a simple, marker-free camera interface with a smart "co-pilot" that understands language and gently corrects mistakes, we can make robots much better at working in messy, real-world factories. It's not a robot that thinks for itself entirely, nor is it a robot that just blindly follows orders. It's a partnership where the human provides the intent ("Go get that valve"), and the robot handles the tricky details of not crashing and landing perfectly.

Technical Summary: From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation

Problem Statement
Teleoperating robotic manipulators in unstructured industrial environments presents significant challenges. Operators must simultaneously manage high-degree-of-freedom arm control, maintain viewpoint and clearance, and avoid collisions with surrounding structures, all while relying on limited depth perception from camera-based interfaces. While pure teleoperation offers flexibility, it imposes a high cognitive burden and risks collisions. Conversely, fully autonomous systems struggle with generalization in dynamic settings due to dependencies on known objects, precise extrinsics, and predefined task parameters. Existing shared-control systems often rely on assumption-heavy target selection (e.g., markers, known object classes) and low-rate collision representations, limiting their safety and responsiveness in real-world industrial scenes.

Methodology
The paper proposes a shared-autonomy framework for mobile manipulation (validated on a Boston Dynamics Spot quadruped) that integrates calibration-free vision-based teleoperation, open-vocabulary target grounding, and user-triggered autonomous execution. The system operates through a modular perception-control pipeline:

Vision-Based Teleoperation Interface:
- Input: A single RGB-D camera (ZED 2i) captures the operator's motion without wearables or fiducials.
- Processing: MediaPipe extracts skeletal landmarks to estimate a body-centered reference frame {B} dynamically. Wrist motion is mapped to the robot's end-effector position, while hand landmarks drive gripper roll and discrete actions (open/close).
- Filtering: A threshold-based rejection filter and exponential moving average (EMA) smooth the trajectory to mitigate depth artifacts and outliers.
- Scaling: The system scales operator arm motion to the robot's workspace based on an online estimation of the operator's arm length, eliminating the need for manual calibration.
Open-Vocabulary Target Grounding:
- Semantic Specification: Operators specify targets using free-form text prompts (e.g., "wheel valve"). A Vision-Language Model (VLM, Qwen3-VL) grounds this prompt on the gripper camera image, returning a bounding box and candidate grasp points.
- Tracking: A promptable video segmentation model (SAM 2) tracks the target across the gripper and body cameras.
- Dynamic Separation: Segmentation masks separate the target object from the static environment within a volumetric map. This ensures the target is excluded from the collision obstacle map, preventing the controller from treating the intended grasp point as an obstacle.
- Grasp Frame: A world-latched grasp frame {G} is established, remaining fixed unless consecutive measurements confirm object movement.
Collision-Aware Reactive Motion Generation:
- Mapping: An onboard GPU-accelerated volumetric mapper (nvblox) fuses stereo depth into a Truncated Signed Distance Field (TSDF) and Euclidean Signed Distance Function (ESDF) for real-time collision checking.
- Control: A Model Predictive Path Integral (MPPI) controller (cuRobo) runs at 50 Hz. It optimizes joint trajectories by minimizing Cartesian tracking error, joint limits, smoothness, and collision costs against the online ESDF.
Shared-Control Assistance:
- Potential Field: Within a 0.4 m activation radius of the grounded target {G}, an attractive potential field biases the operator's commanded reference toward the target.
- Authority Preservation: The correction strength ( $k_{att} = 0.2$ ) is tuned so the operator's command remains the dominant term, allowing them to pull away or override the assistance, while the field corrects fine residual errors during the final approach.
User-Triggered Autonomous Execution:
- Operators can switch to autonomous mode via hand gestures (two fingers to engage, closed fist to execute).
- The system reuses the same collision-aware motion generator to drive the end-effector directly to the grounded frame {G}, inheriting the same safety guarantees as teleoperation.

Key Contributions

Integrated Pipeline: A shared-autonomy system coupling calibration-free teleoperation, open-vocabulary grounding, and gesture-triggered autonomy, converting natural language queries into world-latched 3D grasp frames.
Continuous Collision Awareness: Integration of onboard volumetric mapping with sampling-based MPC, subjecting both teleoperated and autonomous motions to identical collision constraints while dynamically separating the target from the obstacle map.
Potential-Field Assistance: A guidance layer that corrects operator references toward the target within an activation region while maintaining operator authority as the dominant control term.

Results
The framework was evaluated on a quadruped mobile manipulator in industrial valve manipulation and pick-and-place tasks:

Tracking Accuracy: The teleoperation interface achieved a positional Root Mean Square Error (RMSE) of 59 mm relative to motion-capture ground truth.
Collision Avoidance: In cluttered environments, the controller maintained a minimum distance of 18 cm from obstacles even when the operator deliberately commanded the reference 6 cm into the obstacles.
Task Success:
- Full Framework: Achieved 100% success (5/5 trials) in both valve manipulation and pick-and-place tasks.
- Ablation Studies: Removing collision avoidance led to contact failures; removing the potential field led to missed grasps due to alignment difficulties. Both modules were necessary to suppress complementary failure modes.
- Autonomous Mode: Succeeded in 4/5 trials per task, with failures attributed to depth estimation inaccuracies in the grasp service rather than the perception or control pipeline.

Significance and Claims
The paper claims that this framework addresses the gap between the flexibility of teleoperation and the safety of autonomy in unstructured industrial settings. By enabling operators to specify targets via natural language and providing continuous, collision-aware assistance, the system reduces cognitive load and collision risks without requiring wearables, fiducials, or extensive calibration. The authors emphasize that the system successfully handles contact-rich interactions (e.g., turning a valve) and generalizes to different object types (fixed valves vs. movable drills) using a single perception pipeline. The work demonstrates that converting semantic cues into continuous, safety-constrained assistance is feasible for real-time manipulation on legged platforms.