Deep Reinforcement Learning for Humanoid Robot Dribbling

Abstract

Humanoid robot soccer is a very traditional competitive task that aims to push the boundaries of state-of-the-art robotics. One of the many challenges of playing soccer is walking and running while not losing balance. Deep Reinforcement Learning (DRL) has been used to solve complex continuous control problems such as those in robotics. In this work, we focused on learning humanoid robot behavior to dribble a ball against a single opponent. Instead of learning how to control joint commands directly, we adopt an approach where the learning agent interacts with a predefined walking engine. Using DRL model-free algorithms (namely, Deep Deterministic Policy Gradients, Trust Region Policy Optimization, and Proximal Policy Optimization), we effectively learn a high level policy that allows a humanoid robot to fulfill this task. Finally, the learned dribble policy was evaluated on a simulated Nao robot from the RoboCup 3D Soccer Simulation League. According to our results, the learned agent was able to surpass the hand-coded behavior effectively used by the ITAndroids robotics team in the RoboCup competition.