⚡ electrical engineering

A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot

This paper presents a vision-based shared-control teleoperation system that uses an external camera and machine learning to intuitively map an operator's wrist movements to a quadruped robot's manipulator arm while employing a trajectory planner to ensure real-time collision avoidance in hazardous environments.

Original authors: Murilo Vinicius da Silva, Matheus Hipolito Carvalho, Juliano Negri, Thiago Segreto, Gustavo J. G. Lahr, Ricardo V. Godoy, Marcelo Becker

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Murilo Vinicius da Silva, Matheus Hipolito Carvalho, Juliano Negri, Thiago Segreto, Gustavo J. G. Lahr, Ricardo V. Godoy, Marcelo Becker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to control a robotic dog with a mechanical arm, but instead of using a complicated video game controller with joysticks and buttons, you simply use your own hand. That is the core idea of this paper: a new way to "teleoperate" (control from a distance) a four-legged robot using just your movements and a camera.

Here is a breakdown of how it works, using simple analogies:

The Problem: The "Joystick" Struggle

Usually, controlling a robot arm is like trying to paint a masterpiece while wearing thick oven mitts and looking at the canvas through a tiny peephole. You have to use joysticks to move the robot's arm up, down, left, and right. This is hard to learn, tiring for your brain, and easy to mess up, especially if you are in a dangerous place where you can't see everything clearly.

The Solution: The "Ghost Hand"

The researchers created a system where the robot's arm acts like a shadow or a mirror image of your own hand.

The Camera: They use a special camera (an Intel RealSense) that can see depth, like human eyes do. It watches your wrist.
The Magic Link: When you move your wrist forward, the robot's arm moves forward. When you tilt your hand, the robot's hand tilts. It's as if the robot is wearing a "ghost glove" that mimics your exact movements in real-time.

How the Robot "Thinks"

The system isn't just blindly copying you; it has a safety brain built-in.

Collision Avoidance: Imagine the robot arm is a very careful dancer. Even if you move your hand quickly, the robot checks its own body and the surroundings to make sure it doesn't bump into walls or trip over itself. It's like having a personal bodyguard that stops you from walking into a doorframe.
Two Modes of Operation: The system has a clever way to switch between "Manual" and "Auto" without pressing buttons. It uses finger counting:
- Hold up 1 finger: You are in full control. The robot follows your wrist like a loyal dog.
- Hold up 2 fingers: You switch to "Semi-Autonomous" mode. Now, if you make a "fist" gesture over an object (like a soda can), the robot figures out the best way to grab it and does the fine motor work for you. If you open your hand, it lets go.

The Experiment: The "Pick and Place" Test

To prove this works, the team tested it on a real Boston Dynamics Spot robot (the famous robotic dog).

The Task: Two different people had to pick up everyday objects (a chip can and a cleaning bottle) and move them into a box.
The Result: Both users succeeded. They could grab the items, move them, and drop them into the box just by waving their hands.
The Accuracy: They measured how closely the robot followed the human hand. On average, the robot was off by about 7 centimeters (roughly 3 inches). The paper notes that the faster the human moved their hand, the slightly larger the error became, which makes sense because the robot takes a tiny fraction of a second to catch up to your speed.

Why This Matters

This approach is like upgrading from a complex remote control to a natural conversation. You don't need to wear special sensors on your body or learn a new language of buttons. You just use your natural hand movements. It's designed to be a cheaper, easier, and safer way for people to control robots in tricky environments, like factories or disaster zones, without needing to be a robotics expert.

In short: The paper shows that you can control a high-tech robotic arm by simply waving your hand in front of a camera, with the robot smart enough to follow you safely and even help you grab things when you ask it to.

Technical Summary: A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot

Problem Statement
Teleoperating quadruped robots equipped with manipulator arms in hazardous or remote environments presents significant challenges. Traditional control methods relying on joysticks or gamepads often demand high levels of expertise, resulting in steep learning curves and excessive cognitive load for operators. These interfaces struggle to provide intuitive control over high-degree-of-freedom (DoF) manipulators, particularly when operators lack sufficient situational awareness or 3D perception, leading to reduced precision, fatigue, and increased collision risks. While wearable sensors (e.g., IMUs, EMG) offer alternatives, they introduce deployment barriers such as intrusive physical contact, time-consuming calibration, and susceptibility to motion artifacts. Furthermore, monocular vision systems often suffer from depth ambiguity and scale drift, while complex vision pipelines can hinder real-time responsiveness.

Methodology
The authors propose a vision-based shared-control framework that maps human arm movements directly to a robotic manipulator using an external camera, eliminating the need for wearable devices. The system is implemented on a Boston Dynamics Spot quadruped robot with a 6-DoF arm and a jaw gripper.

Motion Capture and Pose Estimation:
- Hardware: An Intel RealSense D435i depth camera is used to acquire synchronized RGB and depth images at 30 Hz.
- Calibration: An ArUco marker establishes a stable reference frame. The system computes the homogeneous transformation matrix between the camera and the marker frame.
- Tracking: MediaPipe Pose is employed to extract 2D wrist coordinates from the RGB stream. Depth values are sampled from the aligned depth image, refined using a median filter over a 5×5 pixel window, and back-projected to obtain 3D wrist coordinates.
- Filtering: An exponential moving average filter ( $\alpha = 0.5$ ) smooths the trajectory, and updates with jumps exceeding 25 cm are discarded to ensure robustness.
- Orientation: Hand landmarks (wrist, index, and little finger metacarpophalangeal joints) are used to compute the palm normal and a roll quaternion, aligning the camera's yaw with the robot's roll axis for end-effector orientation control.
Control Architecture:
- Shared-Control Modes: The system operates in two modes, selected via finger-counting gestures:
  - Manual Teleoperation: The robot's end-effector follows the operator's wrist position in real-time. Gripper actions (open/close) are triggered by open-palm and closed-fist gestures, respectively.
  - Semi-Autonomous Mode: Triggered by a closed-fist gesture over a detected object, the robot autonomously executes a grasp routine. This mode utilizes a YOLOv11-based object detection network to identify manipulable objects and estimate their 3D pose. The system selects the target based on confidence and proximity, then autonomously performs the approach and grasping using the Spot SDK.
- Safety: A trajectory planner continuously checks for collisions with obstacles and the robot's own structure.
Implementation:
- The framework integrates the Boston Dynamics SDK, ROS Noetic, and MoveIt.
- Communication occurs via ROS topics, with the control loop running at approximately 5 Hz for position commands.
- The system includes debugging overlays for visualizing perception and control stages.

Key Contributions

Intuitive Interface: A non-invasive, cost-effective solution using a single depth camera to map human wrist movements to a robotic arm, removing the need for wearable sensors or complex calibration.
Shared-Control Framework: A hybrid architecture that seamlessly switches between direct manual teleoperation and semi-autonomous grasping based on simple hand gestures (finger counting and hand shape).
Robust Perception Pipeline: A method for 3D wrist tracking and hand orientation estimation using monocular depth data (RealSense) combined with MediaPipe, validated against ground truth.
Real-Time Validation: Successful deployment on a real-world quadruped robot (Spot) performing pick-and-place tasks.

Experimental Results
The system was validated through simulation in Gazebo and real-world experiments involving two users performing pick-and-place tasks with everyday objects (a chips can and a cleaning product).

Task Success: Both users successfully grasped and placed objects in real-time, maintaining stable end-effector poses and adjusting gripper orientation as required.
Precision Analysis: Using an Optitrack motion capture system as ground truth, the system achieved a mean positional error of 0.07 meters.
Error Dynamics: The analysis revealed that positional error increases with wrist velocity, attributed to the latency required for the robot's end-effector to reach the target position.
Responsiveness: The system demonstrated the ability to handle continuous control and discrete gesture commands (grasp/release) effectively.

Significance and Claims
The paper claims that this framework provides a cost-effective and accessible alternative for complex telemanipulation, particularly in high-risk environments where traditional interfaces are impractical or too cognitively demanding. By leveraging depth vision and simplified hand tracking, the system offers a balance of intuitiveness, responsiveness, and safety. The authors emphasize that the approach requires neither wearable devices nor traditional interfaces, lowering the barrier to entry for operating mobile manipulators.

The work concludes by modestly noting that while the current system is robust, future iterations will focus on implementing advanced path planning for obstacle avoidance, integrating potential fields for improved shared-control robustness, and conducting comparative studies with traditional interfaces to further validate the system's efficacy.