Eye-Tracking-Driven Control in Daily Task Assistance for Assistive Robotic Arms

Technical Summary: Eye-Tracking-Driven Control in Daily Task Assistance for Assistive Robotic Arms

Problem Statement
Current eye-tracking-driven control systems for assistive robotic arms face significant challenges regarding 3D gaze estimation accuracy and the interpretation of user intent in complex, multi-object environments. Traditional approaches often rely on estimating the user's head position relative to objects, making them susceptible to drift, glasses slippage, and calibration errors. Furthermore, distinguishing between multiple daily tasks and ensuring robust object selection without high-precision 3D gaze tracking remains difficult. Incorrect predictions can lead to frustrating interactions and reduced trust in the system. There is a need for a control framework that reduces reliance on high-accuracy head tracking while enabling individuals with severe physical disabilities to select and execute daily tasks independently.

Methodology
The authors propose a shared-control framework that shifts object localization from the user's perspective to a robot-mounted camera (an "eye-in-hand" configuration). This design eliminates the need for 3D estimation of the user's position, thereby mitigating errors caused by head movement.

• Input Modality: The system utilizes only wearable eye-tracking glasses and a robot-mounted RGB-D camera. Users interact via a dwell-time approach, fixating on task-specific pictograms attached to objects.
• Task Selection: Task pictograms serve as fiducial markers. A YOLOv12n object detection model identifies these pictograms and associated objects. The system supports tasks such as drinking, eating, and switching lights.
• Data Transmission (Feature Matching): To transmit the selected object's data from the eye-tracker scene to the robot camera, the system employs feature matching. The authors evaluated various OpenCV feature detectors (AKAZE, BRISK, ORB, SIFT) and matchers (Brute-Force, FLANN).
  • Algorithm Selection: Through "in-the-wild" testing and comparative analysis, FLANN-AKAZE was selected as the optimal algorithm, offering a balance between the number of matched features, computation time, and robustness against outliers.
• Fallback Mechanism: To handle cases where object classes do not match between the user's view and the robot's view (e.g., due to occlusion or detection errors), a dual-approach fallback system was implemented:
  1. Comparative Approach: Matches bounding box cutouts from both camera views. If classes match, the object is confirmed.
  2. In-the-Wild Search: If the class is missing or mismatched, the system clusters features across the entire robot field of view to locate the area with the highest density of matched features.
• Coordinate Transformation: Once an object is selected, its location is estimated using the known kinematic chain of the eye-in-hand configuration, allowing the robot to plan grasps without needing the user's head coordinates.

Key Contributions

1. Algorithm Comparison: A tailored comparison of feature-matching algorithms for assistive robotic arm data transmission, identifying FLANN-AKAZE as the most suitable for this specific use case.
2. Open-Source Framework: An open-source, eye-tracking-driven robot control framework that resolves head-movement issues by utilizing an eye-in-hand configuration, requiring only standard eye-tracking glasses and a robot-mounted camera.
3. Evaluation and Lessons Learned: A comprehensive evaluation revealing key challenges in feature matching under varying lighting and occlusion conditions. The authors provide specific "lessons learned" regarding the calibration of bounding box resizing, feature matching thresholds, and ratio test parameters to optimize system performance.

Results
The framework was evaluated in a laboratory setting with three specific test cases (SU Cases) involving cups, forks, and bottles under varying conditions (perfect detection, label mismatches, and differing user/robot fields of view).

• Task Selection Success Rate: The system correctly interpreted object and task selection in up to 97.9% of measurements (specifically in SU Case 3.1). The overall mean task selection success rate across evaluations was approximately 95.67%.
• Response Time: The mean time from object selection to message transmission was 260.3 ms (SD 88.5 ms).
• Message Transmission: The "task sent rate" (the percentage of attempts resulting in a transmitted message) varied between 44.9% and 57.8%. The authors note that a lower sent rate is preferable to sending incorrect messages, which could lead to unsafe or unintended robot actions.
• Robustness: The fallback mechanism successfully handled scenarios where object classes were mismatched or occluded, though performance degraded when the user and robot viewed significantly different scenes (SU Case 3).

Significance and Claims
The paper claims that this framework provides a suitable setup for controlling wheelchair-mounted robotic arms by reducing the complexity of the sensory input required. By decoupling object localization from the user's head position, the system offers a more robust solution for real-world daily tasks where head movement is inevitable. The integration of state-of-the-art object detection models (YOLOv12n) and the open-source nature of the framework allow for adaptation to new tasks and objects.

The authors emphasize that while the system demonstrates high accuracy in task selection within controlled environments, it is a foundational step toward real-world application. They acknowledge limitations, such as the current reliance on a limited set of object classes and the need for testing in cluttered, everyday environments. The work is presented not as a final commercial product, but as a modular, adaptable foundation for enhancing independence and safety for users with severe physical disabilities through autonomous robot assistance.