A Deep Reinforcement Learning-Based Decentralized Hierarchical Motion Control Strategy for Multiple Amphibious Spherical Robot Systems With Tilting Thrusters

Abstract

The variable operating conditions and hostile environments faced by underwater robots remain a challenge for motion control in unknown environments. In order to improve the capability of the amphibious spherical robot (ASR) in unknown environments, a decentralized hierarchical deep reinforcement learning (DRL) motion control method based on deep deterministic policy gradient (DDPG) for multiple ASRs system is proposed. In the low-level, a DDPG-based motion controller is trained under a compound rewarding to learn to set the configurations of the tilting angle and rotational speed of each thruster at a proper timescale. At the high-level, a planning controller consisting of different action networks is designed to generate a reasonable thrust target to guide the movement of the robot. Specifically, inspired by the artificial potential field (APF) method, the complex underwater motion can be decomposed into several simple virtual forces. Each action network is trained to learn to generate a virtual thrust target component for a specific action. By combining the outputs of several action networks, the distributed cooperative motion control for multirobot systems can then be easily achieved. Finally, the motion control strategy is applied to the physical multi-ASR system, and the experiment results have shown satisfactory performance.