⚡ electrical engineering

A Novel Neurodynamics-based Approach to Fatigue-aware Motion Planning in Human–Robot Collaborative Assembly

This paper proposes a neurodynamics-based motion planning framework for human–robot collaborative assembly that estimates operator fatigue from facial cues to dynamically adjust robot trajectories and safety margins, thereby balancing task efficiency with enhanced human safety without requiring prior learning.

Original authors: Junfei Li, Jasmun Banwait, Da Long, Simon X. Yang, Sheng Yang

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junfei Li, Jasmun Banwait, Da Long, Simon X. Yang, Sheng Yang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a factory floor where robots and humans are best friends, working side-by-side to build things. Usually, these robot buddies are programmed with a "one-size-fits-all" safety rule: "Stay two feet away from the human, no matter what." But what if the human is super tired? Maybe they are yawning, their eyes are heavy, and they are moving a bit slower. A rigid safety rule doesn't know the difference between a fresh worker and a exhausted one.

This paper suggests a new way for robots to be smarter about safety. Instead of using a fixed rule, the robot uses a "neurodynamics" brain—a special kind of digital nervous system—to read the human's mood and adjust its path in real-time.

The Robot's "Eyes" and "Brain"
First, the robot needs to know if its human partner is tired. It uses cameras to watch the human's face, looking for clues like how wide their eyes are open or if they are yawning. Think of this like a coach watching an athlete; if the athlete's eyes are drooping, the coach knows to slow down the training. The robot uses a mix of computer vision and fuzzy logic (a way of thinking that handles "maybe" and "sort of" instead of just "yes" and "no") to turn these facial clues into a single "fatigue score."

The Digital Nervous System
Once the robot knows the fatigue score, it doesn't just follow a pre-written map. Instead, it uses a "bio-inspired neural network." Imagine the factory floor is covered in a giant, invisible grid of tiny lightbulbs. Each lightbulb represents a spot the robot could go to.

Excitatory Inputs (The Green Light): If a spot is the goal (like where a part needs to be delivered), the lightbulbs there glow bright green, pulling the robot toward them.
Inhibitory Inputs (The Red Light): If a spot is an obstacle or a human, the lightbulbs there glow red, pushing the robot away.

Here is the magic part: The "redness" of the human's lightbulbs changes based on how tired they are.

Low Fatigue: If the human is wide awake, the red light is dim. The robot sees a narrow path between the human and a wall as safe and takes the shortcut to get the job done fast.
High Fatigue: If the human is yawning and tired, the red light gets super bright and spreads out. The robot's "brain" sees the area around the tired human as a big, dangerous zone. It automatically chooses a longer, wider path to give the tired human extra space, prioritizing safety over speed.

What This Approach Rejects
The authors are very clear about what they are not doing. They argue against the old way of doing things, where robots rely on fixed safety margins that never change, or where safety is treated as a separate, static rule that doesn't talk to the planning process. They also reject the idea that robots need to "learn" from years of data to do this. In their system, the robot doesn't need a history lesson; it reacts instantly to the current situation using its neural network, without any training phase.

They also tested this against a popular method called "Artificial Potential Fields" (which is like a robot being pulled by magnets). In their simulations, the old magnetic method got stuck in a "local minimum"—imagine a ball rolling into a small dip in a hill and getting stuck, unable to reach the top. The new neurodynamics method, however, successfully navigated these tricky U-shaped obstacles without getting stuck, showing it handles complex, bumpy environments better.

How Sure Are We?
The paper presents results from computer simulations and real-world experiments.

In the simulations: The robot successfully generated short, efficient paths when the "human" was alert and wider, safer paths when the "human" was tired. It also proved it could escape the "traps" that confused the older magnetic method.
In the real world: The team built a setup with a 3.5m × 2.0m workspace, a mobile robot, and three students assembling circuits. When a student showed signs of fatigue (like yawning), the robot automatically switched to a wider, safer route, avoiding the narrow gap it used when the student was fresh.

The authors suggest that this framework makes robots more "human-centric," meaning they can adapt to the human's state to keep everyone safe and happy. While the results look promising in these specific tests, the paper presents this as a new, effective approach for dynamic environments rather than a solved problem for every possible robot scenario. The system works by constantly updating its "neural activity landscape" to find the best next step, ensuring the robot is always reacting to the human's current state, not a memory of the past.

Technical Summary: A Novel Neurodynamics-based Approach to Fatigue-aware Motion Planning in Human–Robot Collaborative Assembly

1. Problem Statement

Human–Robot Collaborative Assembly (HRCA) aims to combine human flexibility with robotic precision in shared workspaces. However, current HRCA methods exhibit three primary limitations:

Neglect of Internal Human States: Existing studies focus on physical safety and injury prevention but largely ignore internal human states, specifically fatigue. The lack of explicit modeling for fatigue-related cognitive and physical degradation limits a system's ability to adapt assistance strategies, potentially compromising long-term sustainability and safety.
Static Environment Assumptions: Most approaches are designed for static environments with pre-defined task allocations, failing to process dynamic real-world conditions or adapt to evolving human states.
Reactive Safety Mechanisms: Current planning methods often treat safety as static constraints or handle them independently of the planning process. This prevents robots from providing dynamic, proactive protection to operators facing unpredictable motions or evolving task contexts.

The paper addresses the need for a motion planning framework that integrates real-time human fatigue estimation with adaptive robot trajectory generation, ensuring both task efficiency and operator safety.

2. Methodology

The proposed framework integrates vision-based fatigue perception with a bio-inspired neural network (BINN) for motion planning. The methodology consists of four core components:

2.1 Vision-Based Fatigue State Modeling

The system estimates human fatigue levels using a four-stage pipeline:

Data Acquisition & Landmark Extraction: MediaPipe is used to extract facial landmarks. Key geometric metrics, the Eye Aspect Ratio (EAR) and Mouth Aspect Ratio (MAR), are calculated to define Regions of Interest (ROIs).
Feature Extraction: Four complementary indicators are derived:
- Geometric metrics (EAR, MAR).
- Binary states (Eye/Mouth Open or Closed) determined by Convolutional Neural Networks (CNNs) trained on cropped ROIs.
- High-level temporal features: Yawn Duration, Eye Closure Duration, Percentage of Eye Closure (PERCLOS), Eye Amplitude, and Mouth Amplitude.
Fuzzy Logic Inference: A fuzzy inference system maps the extracted features to a quantitative fatigue index. Membership functions classify feature values (low, medium, high), and predefined fuzzy rules (e.g., "If PERCLOS is high, then fatigue is Medium") aggregate these inputs. Defuzzification via the centroid method yields a crisp fatigue value ( $F_t$ ).
Temporal Averaging: To ensure stability and capture sustained trends, the instantaneous fatigue values are averaged over a time window ( $N$ frames) to produce a final fatigue level ( $F_{avg}$ ).

2.2 Neurodynamics-Based Motion Representation

The framework employs a nonholonomic neural network model to represent the robot's configuration space, specifically tailored for car-like mobile robots.

State Space: The neural network is a 3D grid where neurons represent the robot's planar position $(x, y)$ and orientation $\theta$ .
Kinematic Constraints: Unlike point-robot models, the lateral neural connections are restricted to configurations that satisfy the nonholonomic kinematic constraints of the mobile robot (steering and curvature limits). This ensures generated trajectories are physically executable.
Neurodynamic Model: The activity of the $k$ -th neuron ( $x_k$ ) evolves according to a shunting equation:
$\frac{dx_k}{dt} = -Ax_k + (B - x_k)Q^+_k - (D + x_k)Q^-_k$
Where $Q^+_k$ and $Q^-_k$ are excitatory and inhibitory inputs, respectively.

2.3 Fatigue-Aware Planning Mechanism

The planning process dynamically modulates the neural landscape based on the estimated fatigue level:

Excitatory Input ( $Q^+$ ): Driven by the target workspace, creating a global guidance field that directs the robot toward task-relevant regions.
Inhibitory Input ( $Q^-$ ):
- Obstacles: Generate local inhibitory effects that do not propagate.
- Human Operators: The inhibitory effect is modulated by a fatigue-dependent threshold ( $\sigma_c$ ).
- Fatigue Modulation: The threshold $\sigma_c$ is calculated using a sigmoid function of the average fatigue level ( $F_{avg}$ ). As fatigue increases, $\sigma_c$ increases, strengthening the lateral inhibitory propagation around the human's workspace. This effectively enlarges the "safety region" in the neural activity landscape without requiring a learning process.
Path Selection: The robot selects the next configuration by choosing the neighboring neuron with the highest neural activity, ensuring the path is collision-free and kinematically feasible.

3. Key Contributions

The paper outlines three primary contributions:

Fatigue-Modulated Inhibitory Mechanism: Unlike treating fatigue as a fixed safety rule, the estimated fatigue level is integrated directly into the inhibitory inputs of the BINN. This allows the safety margin around human work regions to adapt dynamically to the operator's state.
Nonholonomic Neural Connection Design: A specific neural connection strategy is developed for car-like mobile robots. Connections are established only between configurations satisfying kinematic and steering constraints, ensuring the generated motion is restricted to physically executable transitions rather than idealized point-robot motions.
Fatigue-Aware Safety-Efficiency Trade-off: The framework enables a dynamic trade-off. In low-fatigue states, the robot selects shorter, efficient paths. In high-fatigue states, the system automatically increases clearance from the operator, prioritizing safety and comfort over path length.

4. Results

The framework was evaluated through simulation and real-world experiments:

Simulation (Different Fatigue Levels):
- Low Fatigue: The robot generated a short, efficient trajectory through a narrow passage between an obstacle and a human work region.
- High Fatigue: The increased inhibitory threshold caused the narrow passage to become less desirable in the neural landscape. The robot successfully generated an alternative trajectory with larger clearance, avoiding the fatigued operator.
- Comparison with APF: In a concave U-shaped environment, the proposed method successfully navigated the robot out of the concave region. In contrast, the Artificial Potential Field (APF) method failed, trapping the robot in a local minimum due to the cancellation of attractive and repulsive forces.
Real-World Experiments:
- Setup: A ROS-based mobile robot operated in a shared workspace with three human students assembling circuits. Cameras monitored facial data for fatigue estimation.
- Scenario 1 (Low Fatigue): The robot efficiently delivered components, navigating through narrow gaps between obstacles and operators.
- Scenario 2 (High Fatigue): When one student exhibited high fatigue (e.g., yawning, eye closure), the system detected the state and adjusted the robot's path. The robot avoided the narrow passage near the fatigued student, selecting a longer but safer route with increased clearance.

5. Significance and Claims

The authors claim that the proposed neurodynamics-based framework supports the development of safer, more human-centric robotic systems in collaborative assembly. The significance of the work lies in:

Adaptability: The system responds to dynamic human internal states (fatigue) without requiring offline learning or retraining.
Safety-Efficiency Balance: It provides a mechanism to dynamically balance task efficiency with operator safety, expanding safety margins only when necessary.
Robustness: The neurodynamic approach demonstrates superior performance in complex environments with concave obstacles compared to traditional potential field methods, avoiding local minima.
Practicality: The integration of vision-based fatigue estimation with real-time motion planning offers a viable solution for Industry 5.0 scenarios where robots act as adaptive, human-centric collaborators.

The paper concludes that this approach effectively addresses the limitations of current HRCA methods by explicitly modeling human internal states and integrating them into a unified, adaptive planning framework.