Developing basic soccer skills using reinforcement learning for the RoboCup small size league

Abstract

This study has started as part of a research project at Stellenbosch University (SU) that aims at building a team of soccer-playing robots for the RoboCup Small Size League (SSL).In the RoboCup SSL the Decision-Making Module (DMM) plays an important role for it makes all decisions for the robots in the team.This research focuses on the development of some parts of the DMM for the team at SU.A literature study showed that the DMM is typically developed in a hierarchical structure where basic soccer skills form the fundamental building blocks and high-level team behaviours are implemented using these basic soccer skills.The literature study also revealed that strategies in the DMM are usually developed using a hand-coded approach in the RoboCup SSL domain, i.e., a specific and fixed strategy is coded, while in other leagues a Machine Learning (ML) approach, Reinforcement Learning (RL) in particular, is widely used.This led to the following research objective of this thesis, namely to develop basic soccer skills using RL for the RoboCup Small Size League.A second objective of this research is to develop a simulation environment to facilitate the development of the DMM.A high-level simulator was developed and validated as a result.The temporal-difference value iteration algorithm with state-value functions was used for RL, along with a Multi-Layer Perceptron (MLP) as a function approximator.Two types of important soccer skills, namely shooting skills and passing skills were developed using the RL and MLP combination.Nine experiments were conducted to develop and evaluate these skills in various playing situations.The results showed that the learning was very effective, as the learning agent executed the shooting and passing tasks satisfactorily, and further refinement is thus possible.iv