Reinforcement Learning Integrated Active Force Control for Five-link Biped Robots

Abstract

In this paper, we present a novel control model that combines reinforcement learning (RL) and active force control (AFC) for the trajectory tracking of biped robots. The AFC-RL controller is crafted to harness the robustness of RL and the simplicity of AFC in trajectory tracking. Implemented on a 5-link biped robot, the controller aims to enhance trajectory tracking and disturbance rejection capabilities. To estimate the inertia matrix of AFC in real-time, an RL module is trained using feedback tracking errors from the biped robot. We employ a deep deterministic policy gradient algorithm coupled with two reward schemes—one based on duration and the other on trajectory-following accuracy. These reward schemes synergistically enable RL to estimate the inertia matrix of AFC while minimizing errors. With an accurate inertia matrix estimation, the AFC-RL output torques are controlled to achieve precise trajectory tracking for the biped robot. For performance evaluation, we developed a five-link biped robot model in MATLAB/Simulink and conducted simulations with various routes and external disturbances. Our AFC-RL controller is compared with an existing AFC employing an iterative learning (IL) scheme from the literature. Across all scenarios, AFC-RL consistently outperforms AFC-IL in terms of average tracking error (ATE). Notably, in the case of varying sinusoid disturbances, AFC-RL surpasses AFC-IL by 2.25% in ATE. The evaluation results affirm the effectiveness of the proposed AFC-RL model for trajectory tracking and disturbance rejection in controlling biped robots.