Walking Control of a Biped Robot on Static and Rotating Platforms Based on Hybrid Reinforcement Learning

Abstract

In this paper, we proposed a novel Hybrid Reinforcement Learning framework to maintain the stability of a biped robot (NAO) while it is walking on static and dynamic platforms. The reinforcement learning framework consists of the Model-based off-line Estimator, the Actor Network Pre-training scheme, and the Mode-free on-line optimizer. We proposed the Hierarchical Gaussian Processes as the Mode-based Estimator to predict a rough model of the system and to obtain the initial control input. Then, the initial control input is employed to pre-train the Actor Network by using the initial control input. Finally, a model-free optimizer based on Deep Deterministic Policy Gradient framework is introduced to fine tune the Actor Network and to generate the best actions. The proposed reinforcement learning framework not only successfully avoids the distribution mismatch problem while combining model-based scheme with model-free structure, but also improves the sample efficiency for the on-line learning procedure. Simulation results show that the proposed Hybrid Reinforcement Learning mechanism enables the NAO robot to maintain balance while walking on static and dynamic platforms. The robustness of the learned controllers in adapting to platforms with different angles, different magnitudes, and different frequencies is tested.