LEARNING TO CONTROL THE THREE-LINK MUSCULOSKELETAL ARM USING ACTOR–CRITIC REINFORCEMENT LEARNING ALGORITHM DURING REACHING MOVEMENT

Abstract

Learning to control the planar three-link musculoskeletal arm by using an Actor–Critic learning algorithm during reaching movements to stationary target is presented. The arm model used in this study includes three skeletal links (hand, forearm and upper arm), three joints (wrist, elbow and shoulder without redundancy) and six nonlinear monoarticular muscles with redundancy which are modeled based on Hill model. The learning system is composed of Actor and Critic parts. For each part, a single layer neural network is used. This learning system applies six activation commands to six muscles at each instant of time. It also uses a reinforcement (reward) feedback for learning process and controlling the arm movement direction. The results showed that with a learning rate α = 0.9 and after 20 episodes, Mean square error (MSE), average reward and average time of reaching the target are gradually converged to the values: 0.0056, 0.02262 and 187 s, respectively. After the 20th episode, the learning will be completed. The research suggests a new direction for designation of learning-based controllers for functional electrical stimulation (FES) applications and for arm movement of autonomous robots.