How Should We Teach Robots? A Comparison of Kinesthetic, Joystick, and Gesture-Based Teaching

Technical Summary: A Comparison of Kinesthetic, Joystick, and Gesture-Based Robot Teaching

Problem Statement
While Learning from Demonstration (LfD) and Programming by Demonstration (PbD) allow non-expert users to teach robots through examples rather than code, the choice of teaching modality significantly impacts the quality of the resulting demonstration and the user's experience. Existing literature has compared specific modalities in isolation or focused on generalization; however, there is a need for a direct, user-centered comparison of kinesthetic guidance, joystick teleoperation, and hand gestures on identical physical manipulation tasks. This study addresses the trade-offs between usability, demonstration efficiency, and replay success across these three modalities, particularly in scenarios where direct physical contact with the robot may be unsafe, restricted, or physically demanding.

Methodology
The authors conducted a user study involving eight participants (aged 24–40) who taught a Franka Robotics Panda robot three distinct manipulation tasks using the Robothon Electronic Task Board. The tasks were:

1. Peg Pick: Picking a peg and placing it in a bowl (primarily top-down grasping).
2. Probe Measure: Picking a probe, touching a specific measurement point, and placing it in a bowl (requiring precise orientation).
3. Cable Wrap: Wrapping a cable around a fixture (requiring continuous, contact-rich motion).

Each task was presented in multiple scene variants (e.g., different locations or obstacles) to elicit adapted demonstrations. The study evaluated three teaching modalities:

• Kinesthetic Guidance: Users physically guided the robot arm with gravity compensation enabled.
• Joystick Teleoperation: Users controlled the robot via a gamepad, decomposing motion into sequential translation and orientation commands.
• Hand Gestures: A contact-free 6-DoF teleoperation method using a Leap Motion sensor. This utilized a "clutch-based" interaction where closing the hand engaged control (following 6-DoF motion) and opening the hand disengaged it. Specific gestures controlled end-effector rotation.

Evaluation Metrics
The study analyzed a dataset of 168 recorded demonstrations (resulting in over 600 execution roll-outs) using the following metrics:

• Replay Success: Whether the robot successfully achieved the task goal when replaying the recorded demonstration.
• Demonstration Duration: Used as an objective proxy for teaching effort and efficiency.
• Subjective Workload: Measured via a modified NASA-TLX questionnaire assessing mental demand, physical demand, effort, frustration, and performance satisfaction.
• Error Analysis: Classification of common failures (e.g., slip grasps, imprecise alignment, loss of hand tracking) and trajectory characteristics (smoothness vs. segmentation/noise).

Key Results

• Replay Success: Kinesthetic guidance generally yielded the highest success rates, particularly for orientation-sensitive and contact-rich tasks (Probe Measure: 83.0%; Cable Wrap: 58.1%). Joystick teleoperation performed best on the simple Peg Pick task (99.2% success), outperforming both kinesthetic and gesture methods. Hand gestures achieved competitive results on Peg Pick (92.7%) but struggled with the most complex tasks.
• Efficiency (Duration): Kinesthetic teaching was consistently the fastest modality (17.4s – 24.0s average). Joystick and gesture demonstrations were significantly longer, especially for tasks requiring fine orientation (e.g., Cable Wrap took ~65s for joystick and ~75s for gestures).
• Workload: Kinesthetic teaching resulted in the lowest mental demand, effort, and frustration, and the highest performance satisfaction. Joystick teleoperation had the lowest physical demand but higher mental demand and frustration due to the cognitive load of decomposing motions. Hand gestures showed workload levels similar to joysticks but with higher variability between users.
• Error Patterns: Failures were primarily manipulation errors (e.g., slips, misalignments) rather than task misunderstandings. Joystick errors often stemmed from the need to decompose motion into discrete steps. Gesture errors were frequently caused by hand-tracking instability and noise during fine orientation control.

Significance and Claims
The paper claims that while kinesthetic teaching remains the most efficient and precise baseline for demonstration acquisition when physical access is safe and available, it is not universally superior. The study demonstrates that:

1. Task Dependency: The optimal modality is highly dependent on the task structure. Joystick teleoperation is highly effective for simple, translational tasks (like top-down grasping), while kinesthetic guidance is superior for tasks requiring precise alignment or continuous contact.
2. Viability of Contact-Free Methods: Hand-gesture teaching, despite being less reliable overall due to tracking noise, performed better than expected. It serves as a viable contact-free alternative for scenarios where kinesthetic teaching is impossible, unsafe, or physically demanding, particularly for simpler tasks.
3. User Experience: The study highlights that the "naturalness" of a modality does not always correlate with efficiency; while gestures felt natural to some, the cognitive load of managing tracking errors increased frustration and demonstration time.

The authors conclude that robot teaching strategies should be modality-dependent, selecting the interface based on the specific constraints of the workspace (safety/access) and the complexity of the manipulation task (orientation vs. translation requirements). The study acknowledges limitations regarding its small sample size and single-robot platform, framing its findings as exploratory comparisons intended to guide future research with larger cohorts and diverse task types.