💻 computer science

NPU-Accelerated Neuro-Fuzzy Safety Assistance with Decision-Level Multimodal Fusion for Teleoperated Assistive Robots

This paper presents an Edge-AI shared-control framework for teleoperated assistive robots that utilizes a Hailo-8L NPU to concurrently execute an Adaptive Neuro-Fuzzy Inference System and a YOLOX-S detector, fusing their outputs to significantly reduce collision rates and emergency interventions while maintaining full local operation.

Original authors: Sarra Ben Halima, Faten Ben Abdallah, Joseph Haggege

Published 2026-09-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sarra Ben Halima, Faten Ben Abdallah, Joseph Haggege

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that helps a person move around a room, but the robot does not decide where to go. Instead, a human sits at a remote control station, watching a video feed and steering the robot with a joystick. This setup is common for tasks like checking on elderly relatives or inspecting dangerous areas, but it has a dangerous flaw. The human operator sees only what the camera shows, which is often a narrow slice of the room. They might not see a glass door, a low table, or a person standing just outside the camera's view. If the operator pushes the robot forward, it might crash before the human even realizes the danger is there. The solution is not to make the robot drive itself, which would remove the human's control and require complex, expensive mapping systems. The solution is to let the human steer while a smart safety layer watches the path and gently slows the robot down if something is too close.

Researchers at the National Engineering School of Tunis have built and tested exactly this kind of system for small, wheeled robots. Their goal was to create a safety net that works entirely on the robot itself, without needing to send data to a cloud server or rely on a powerful computer. They combined two different ways of seeing the world. First, the robot uses infrared sensors that act like a blind person's cane, bouncing signals off objects to measure distance. Second, it uses a camera and a visual recognition system to identify what those objects are. The researchers taught a computer program to blend these two streams of information. The program learns to slow the robot down based on how far away an object is, but it also listens to the camera. If the camera sees a person or a chair with high confidence, the program reduces the robot's speed even more, acting as a second pair of eyes that never blinks.

To make this system fast enough to be useful, the team faced a significant challenge. Running complex vision software and safety calculations at the same time usually requires a heavy, power-hungry computer. Instead, they used a specialized chip designed for artificial intelligence, called a neural processing unit, which is small enough to fit on a standard single-board computer. This chip allowed the robot to process visual data and make safety decisions simultaneously, updating its speed fifty times every second. The system does not plan a route or choose a destination; it simply takes the direction the human operator chooses and decides how fast the robot is allowed to travel in that direction. If the robot gets too close to an obstacle, the system overrides the human's command to move forward and forces the robot to reverse, ensuring a safe stop.

In their experiments, the team tested this system on a custom-built four-wheeled robot in a controlled environment. They asked a human operator to drive the robot through a course filled with obstacles under three different conditions: driving with no help, driving with the infrared sensor safety system, and driving with the combined infrared and camera system. The results showed a clear improvement in safety. When the operator drove without assistance, the robot collided with obstacles in forty percent of the attempts. When the combined safety system was active, that collision rate dropped to just ten percent. Furthermore, the robot maintained a much safer distance from obstacles, keeping a gap of about ten centimeters instead of the five centimeters seen in the unassisted runs. The system also reduced the number of times the human operator had to hit the emergency stop button, dropping from an average of nearly two interventions per run to just one-fifth of an intervention.

The researchers found that the system worked smoothly without slowing down the robot's ability to respond to the human's steering commands. The entire process, from seeing an obstacle to adjusting the speed, happened in less than a hundredth of a second. The visual system was particularly effective at detecting objects that the distance sensors might miss, such as low-contrast surfaces, and it adjusted the robot's speed based on how confident it was about what it saw. However, the team noted that the system is not perfect. The safety rules were tested in a laboratory with static objects and a single operator, so the results might vary in a busy home with moving people. Additionally, if the visual pipeline stalls or a frame is delayed, the system discards the outdated visual data and reverts to relying only on the distance sensors, rather than detecting a specific camera blockage. Despite these limitations, the study demonstrates that it is possible to give a teleoperated robot a reliable, local safety layer that protects against collisions without taking away the human's control.

This work matters because it offers a practical path forward for assistive robots that need to be affordable and safe. By proving that complex safety calculations can run on a small, low-power chip, the researchers showed that these robots do not need to be tethered to the internet or require expensive hardware to be safe. The system respects the human operator's intent while adding a layer of protection that is always watching. In the future, such technology could allow elderly people to use remote-controlled robots to fetch items or monitor their homes without the constant fear of the machine crashing into furniture or people. The researchers plan to test the system with more operators and in more dynamic environments to see how it holds up when the world is less predictable than a laboratory. For now, they have shown that a robot can be both obedient to a human and smart enough to protect itself.

Technical Summary: NPU-Accelerated Neuro-Fuzzy Safety Assistance with Decision-Level Multimodal Fusion for Teleoperated Assistive Robots

Problem Statement
Teleoperated mobile robots used in assistive and domestic monitoring tasks face a fundamental trade-off: purely manual teleoperation places the entire burden of collision avoidance on the human operator, whose situational awareness is often limited by narrow camera views and weak depth cues. Conversely, fully autonomous navigation removes operator intent and requires computational resources (mapping, localization, planning) that exceed the cost, size, and power budgets of low-cost domestic platforms. Existing shared-control solutions often rely on brittle PID controllers or fuzzy logic systems that lack learning capabilities, or they treat perception and control as independent modules, failing to leverage synergies between deep learning and neuro-fuzzy control on edge hardware. Furthermore, most decision-level fusion strategies for safety have not been integrated with dedicated Neural Processing Unit (NPU) acceleration for concurrent execution.

Methodology
The authors propose a shared-control framework where the human operator retains full steering authority (direction), while an onboard Edge-AI safety layer automatically regulates the admissible translational velocity. The system architecture is divided into three layers:

Sensing Layer: Utilizes a custom 360° servo-scanned dual-IR ranging system (two Sharp GP2Y0A41SK0F sensors) for geometric distance and bearing measurements, an IMU for attitude estimation via a 6-axis Extended Kalman Filter (EKF), and an RGB camera for visual perception.
Low-Level Embedded Layer: An STM32L4S5 handles attitude estimation, while an Arduino MEGA 2560 manages motor actuation and serial communication.
High-Level Assistance Layer: Executing on a Raspberry Pi 5 equipped with a Hailo-8L NPU, this layer runs two concurrent inference pipelines:
- ANFIS Controller: An Adaptive Neuro-Fuzzy Inference System (Sugeno-type, 20 rules) that maps IR distance and bearing to a base velocity ( $v_{base}$ ).
- YOLOX-S Detector: A pretrained anchor-free object detector that provides a semantic obstacle confidence score ( $s_{obs}$ ) from the visual stream.

Decision-Level Multimodal Fusion
The core innovation lies in a lightweight, multiplicative decision-level fusion law that combines the geometric and semantic branches:
$\bar{v}_f = \bar{v}_A (1 - s_{obs})$
Where $\bar{v}_A$ is the normalized ANFIS output and $s_{obs}$ is the maximum confidence of obstacles detected in the robot's frontal region. This formulation attenuates the velocity based on visual confidence without requiring complex weight calibration. Additionally, a deterministic critical-distance override ( $d \leq 5$ cm) operates independently of the learned models, forcing a reverse maneuver to ensure safety even if the learned components fail or the operator commands forward motion.

NPU Deployment
The system achieves real-time performance by deploying both the ANFIS controller and the YOLOX-S detector concurrently on a single Hailo-8L NPU. The ANFIS model was converted from PyTorch to ONNX, quantized to INT8, and compiled into a Hailo Executable Format (HEF) binary. This allows for 50 Hz ANFIS updates and 28.7 fps visual inference with a combined NPU utilization of 55%. The entire stack operates locally without cloud dependency.

Key Contributions

Shared-Control Architecture: A system where the operator controls direction while an Edge-AI layer regulates speed based on obstacle configuration.
Dual-Inference on Single NPU: Successful concurrent execution of a neuro-fuzzy controller and a deep learning detector on a single Hailo-8L NPU, with measured latency and throughput under joint load.
Decision-Level Fusion: A multiplicative fusion mechanism that modulates geometric velocity with visual confidence, allowing the system to attenuate speed based on semantic detection without merging raw sensor data.
Independent Safety Override: A critical-distance rule that prioritizes geometric safety over both learned components and operator commands.
Deployment Pipeline: A complete workflow converting a PyTorch-trained ANFIS model to an INT8-deployed HEF binary, including a cost-effective 360° servo-scanned IR ranging system.
Experimental Validation: Comparative trials evaluating manual, ANFIS-assisted, and ANFIS–YOLOX-assisted teleoperation.

Experimental Results
Experiments were conducted on a custom four-wheeled platform with a single operator over 10 runs per condition (Manual, ANFIS-only, Fusion). Key findings include:

Safety Metrics: The fusion strategy reduced the collision rate from 0.40 (Manual) to 0.10 collisions/run. Minimum obstacle clearance increased from 5.1 cm to 10.4 cm, and emergency interventions dropped from 1.70 to 0.20 per run.
Performance: The system achieved 50 Hz ANFIS updates (0.62 ms latency on NPU) and 28.7 fps YOLOX inference. The INT8 deployment showed a deviation of only 0.4% relative to the FP32 baseline.
Trade-offs: The safety improvements came at the cost of a moderate increase in command-to-actuation latency (+6.2 ms) and course completion time (+20.3%), reflecting a more conservative velocity profile.

Significance and Claims
The paper claims that this architecture addresses the gap in co-designing neuro-fuzzy control, deep-learning perception, and NPU acceleration for low-cost assistive robots. By executing both control and perception on a single edge accelerator, the system demonstrates that shared-control safety layers can be implemented within strict power and size constraints without cloud dependency. The authors emphasize that the system does not replace operator intent but enhances safety by dynamically regulating speed based on real-time geometric and semantic data.

Limitations and Future Work
The authors explicitly note several limitations:

The ANFIS model was trained offline on controlled data and exhibits non-monotonic behavior near the lower distance bound, necessitating the independent critical-distance override.
The YOLOX detector was used as a pretrained component without independent benchmarking (precision/recall) on the specific deployment environment.
The system relies on a single operator and static obstacles; inter-operator variability and dynamic environments were not tested.
The reverse override does not check for rear obstacles, and the IR ranging accuracy at short distances was not quantified against reference standards.

Future work is proposed to quantify IR ranging errors, benchmark the visual detector, impose monotonicity constraints on the ANFIS surface, and validate the system across multiple operators and dynamic scenarios.

Technical Summary: NPU-Accelerated Neuro-Fuzzy Safety Assistance with Decision-Level Multimodal Fusion for Teleoperated Assistive Robots

More like this