Natural Walking With Musculoskeletal Models Using Deep Reinforcement Learning

Abstract

Human gait optimality has been investigated recently, with the development of detailed musculoskeletal models, through trajectory optimization approaches or deep reinforcement learning (DRL). Trajectory optimization studies are limited by the trajectory length and can only generate open-loop solutions. While existing DRL solutions provide closed-loop control policies without trajectory length limit, they either do not evaluate the naturalness of the behaviour, or directly impose experimental tracking data. In this letter, a DRL-based approach is proposed with a nature-inspired curriculum learning (CL) scheme and a neuromechanically-inspired reward function. This approach generates close-to-natural human walking without the aid of experimental data. Our CL scheme is realized by an evolving reward function, targeting simpler behaviours such as standing and stepping first, then gradually refining the gait. The emerged gait from the closed-loop stochastic policy demonstrated a strong correlation with human gait kinematics, with Pearson correlations of 0.95 and 0.83 at the hip and knee, respectively, and higher gait symmetry than two other DRL-based control policies without CL. Our approach was also found to have efficient convergence to walking-capable policy. This approach can facilitate the development of assistive robotic systems by providing a “human” controller, and could enable decentralized adaptation between the agent and the assistive robotic devices.