Developing Hexapod Locomotion Through Reinforcement Learning

Abstract

This paper presents an effort to develop self-learned hexapod robot locomotion through reinforcement learning (RL). A comparison of reinforcement learning algorithms—Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Soft Actor-Critic (SAC)—is carried out for training the hexapod to travel to goals in randomized positions. The comparison is conducted in a physics-based simulation environment. Our experiments compare algorithms based on accumulated rewards. Results indicate that SAC outperforms PPO and DDPG during training cycle. However, PPO performed best during model evaluation. We use the results as a basis for providing guidelines for the selection of an appropriate algorithm for legged robot control tasks. We then present suggestions for further investigations that could provide value to reinforcement learning when applied to legged robotics.