Musculoskeletal Motion Imitation for Learning Personalized Exoskeleton Control Policy in Impaired Gait

1. Problem Statement

Designing generalizable control policies for lower-limb exoskeletons is currently hindered by two main bottlenecks:

• Data Intensity and Time: Traditional approaches rely on "human-in-the-loop" optimization, which requires exhaustive, iterative physical trials with individual users. This is time-consuming, labor-intensive, and physically demanding, making it impractical for clinical populations who may have limited physical tolerance or high fall risks.
• Lack of Generalization: Existing deep learning controllers often require massive biomechanical datasets collected in high-fidelity labs and struggle to generalize across diverse locomotor tasks or specific pathological conditions (e.g., muscle weakness) without retraining.
• Device Specificity: Many existing simulation-based controllers are tuned for specific hardware, limiting their applicability to different exoskeleton configurations.

The authors propose a solution that eliminates the need for extensive physical trials by leveraging physiologically plausible musculoskeletal simulation combined with Reinforcement Learning (RL) to generate personalized, device-agnostic control policies for both able-bodied and impaired gait.

2. Methodology

The framework operates in three distinct phases within a high-fidelity simulation environment:

A. Simulation Environment

• Engine: Built on Hyfydy (a high-fidelity physics engine) with the SCONE Python API.
• Model: Utilizes the Hyfydy 3D H2190 model, featuring 21 degrees of freedom and 90 muscles. The model includes realistic physiological properties such as tendon elasticity, muscle pennation, and fiber damping.
• Exoskeleton Integration: Ideal torque motors are added to bilateral hip, knee, and ankle joints. Device mass and inertia are distributed across limb segments based on literature values to simulate realistic energy expenditure.

B. Learning Framework (Deep Reinforcement Learning)

The system uses the Soft Actor-Critic (SAC) algorithm to train a neural network policy ($\pi$).

• Observation Space ($S$): Includes current physiological states (muscle fiber length, force, activation, ground reaction forces, joint angles) and future reference kinematics (joint angle changes over the next 4 timesteps) to enable motion imitation.
• Action Space ($A$):
  • Base: 90 muscle excitations mapped to $[0, 1]$.
  • Fine-tuning: Extended to include 6 normalized torque outputs for bilateral hip, knee, and ankle joints.
• Reward Function: A weighted sum of eight terms designed to balance tracking fidelity with physiological plausibility:
  • Kinematic/Kinetic Tracking: Position, velocity, root trajectory, end-effector (feet/head), and joint torque matching against reference data.
  • Physiological Constraints: Metabolic energy minimization (using Umberger/Uchida models), muscle excitation smoothness, and exoskeleton energy penalty (to prevent excessive torque usage).

C. Training Pipeline

1. Base Model Training: The agent learns to imitate able-bodied human locomotion (walking and running at 0.6–2.2 m/s) using motion capture data as a reference. The goal is to reproduce both kinematics and kinetics (joint moments).
2. Exoskeleton Fine-Tuning (Able-Bodied): The base policy is fine-tuned to minimize metabolic cost by activating the exoskeleton torque actuators. The reward weights for tracking are slightly reduced, and the metabolic penalty is increased to encourage energy efficiency.
3. Impaired Gait Simulation & Policy Generation:
   • Impairment Modeling: Specific muscle groups (e.g., plantarflexors, hip flexors) are artificially weakened by reducing their maximum excitation to 5% of baseline.
   • Compensatory Learning: The impaired model is trained to imitate the healthy reference gait using the exoskeleton. The hypothesis is that the RL agent will naturally learn to generate asymmetric, deficit-specific assistive torques to compensate for the weakness and restore healthy gait patterns.

3. Key Contributions

1. Physiologically Plausible Imitation: The framework successfully reproduces not just kinematics but also internal kinetics (joint moments) and ground reaction forces, validating the physiological realism of the learned agent.
2. Device-Agnostic Control: The learned policies are not tied to specific hardware parameters, allowing for scalable application to different exoskeleton configurations (demonstrated here for hip and ankle).
3. Automated Impaired Gait Modeling: The system simulates specific muscular deficits and automatically generates compensatory gait patterns (e.g., hip-hiking for hip flexor weakness) that match clinical observations.
4. Personalized Assistance without Human Trials: The framework generates optimized, asymmetric assistive torque profiles for impaired users entirely in simulation, eliminating the need for iterative physical optimization.

4. Results

A. Able-Bodied Performance

• Tracking Fidelity: The base model achieved high-fidelity tracking with low Root Mean Square Errors (RMSE) for joint angles (2.85°–4.03°) and joint moments (0.10–0.11 Nm/kg). It also reproduced Ground Reaction Forces (GRF) with an RMSE of 0.14 body weights, despite GRF not being a direct training input.
• Metabolic Reduction: The learned exoskeleton policies reduced gross metabolic cost compared to the no-exoskeleton baseline:
  • Hip Exoskeleton: 4.57% reduction (walking), 6.95% (running).
  • Ankle Exoskeleton: 6.15% reduction (walking), 13.81% (running).
• Torque Profiles: The generated torque profiles aligned closely with state-of-the-art profiles validated in human experiments.

B. Impaired Gait Performance

• Compensatory Strategies: The simulated impaired models exhibited clinically recognized compensatory behaviors, such as excessive dorsiflexion for plantarflexor weakness and "hip-hiking" (pelvic tilt) for hip flexor weakness.
• Targeted Asymmetric Assistance: The learned policies generated asymmetric torque profiles specifically targeting the affected limb:
  • Plantarflexor Weakness: Peak plantarflexion assistance on the affected side was 59% higher than the non-affected side.
  • Hip Flexor Weakness: Peak hip flexion assistance on the affected side was 49% higher.
• Outcomes:
  • Metabolic Cost: Reduced by 8.1% (plantarflexor weakness) and 13.1% (hip flexor weakness).
  • Symmetry: Bilateral kinematic symmetry improved significantly, with RMSE between affected and non-affected joints decreasing by ~40–50%.

5. Significance and Conclusion

This study demonstrates that Reinforcement Learning within high-fidelity musculoskeletal simulation is a viable, scalable alternative to physical human-in-the-loop optimization.

• Clinical Impact: It offers a pathway to personalized exoskeleton control for patients who cannot undergo extensive physical trials due to fatigue, pain, or fall risk.
• Efficiency: It drastically reduces the time and cost associated with controller design by shifting the optimization loop entirely to simulation.
• Future Work: While the simulation results are promising, the authors acknowledge the need for Sim-to-Real transfer. Future steps involve deploying these learned policies onto physical hardware and validating them with clinical populations to confirm the reduction in metabolic cost and improvement in gait symmetry in real-world settings.

In summary, the paper presents a unified computational model that bridges the gap between healthy and pathological gait, using simulation to design assistive devices that are both energetically efficient and kinematically symmetrical for diverse user populations.