Balance Control of a Bipedal Robot Utilizing Intuitive Pattern Generators with Extended Normalized Advantage Functions

Abstract

Herein, a combination of Local Pattern Generators (LPG) with reinforcement learning is proposed to balance a bipedal robot using minimal power consumption. This work presents the extension of Normalised Advantage Function (eNAF) algorithm to work with recurrent neural networks without sacrificing time-dependency between data in the same episode. Additionally, a hybrid controller is introduced by combining eNAF algorithm hierarchically with LPGs to provide more robustness with less computational power requirements. The system was asynchronous, as pattern generator had an activation frequency of 100Hz, while eNAF algorithm had only 1Hz and were not synchronised between them. Robot autonomy time was increased through reduction of computational load by introducing variable-ratio activation frequency between the LPGs and the eNAF algorithm. Finally, a new and novel bipedal robot design with non-conventional linear actuators was used as the basis of the simulator model. These experiments were implemented using V-Rep Edu simulator with the industrial Vortex Studio dynamic engine. The results demonstrate a fast and agile recovery by the trained robot after a push in transverse plane.