Efficient exploration by switching agents according to degree of convergence of learning on Heterogeneous Multi-Agent Reinforcement Learning in Single Robot

Abstract

In recent years, a robot is required to perform autonomously in complex environment. Some researchers use reinforcement learning that learns actions autonomously according to environment. Reinforcement learning requires exploratory actions, but in conventional reinforcement learning it was random. Random exploratory actions are inefficient and takes a lot of time to learn. To prevent inefficient exploratory actions, we proposed a method that uses Heterogeneous Multi-Agent Reinforcement Learning system (HMARL) in previous research. HMARL enables efficient exploratory actions by using multiple agents with heterogeneous learning spaces. HMARL system is a system that performs exploratory actions using the learning of multiple agents. In addition, HMARL needs an index that autonomously selects an agent from among all the agents inside heterogeneous learning space. We propose a method to select an agent using the degree of convergence of the learning of the agents in HMARL based on the TD errors. As a result, efficient exploratory actions by multiple agents with different learning spaces was achieved. Then, experiment to compare the proposed method and the method of previous research was conducted. From experimental results, the usefulness of the proposed method has been demonstrated.