Automatic Curriculum Determination for Deep Reinforcement Learning in Reconfigurable Robots

摘要

Deep reinforcement learning (DRL) is a prevalent learning method in robotics. DRL is commonly applied in real-world scenarios such as learning motion behavior in rough terrain. However, the lengthy learning epochs reduce DRL practicability in many such environments. Curriculum learning can significantly enhance the efficiency of DRL, but establishing a curriculum is challenging, partly because it can be difficult to assess the operation complexity for each task. Determining operation complexity can be especially difficult for reconfigurable search and rescue robots. We present a method for learning based on an automatically established curriculum tuned to the robot’s perspective. The method is especially suitable for outdoor environments with multiple obstacle variants, e.g., environments encountered in search and rescue missions. After an initial learning stage, the behavior of a robot when overcoming each obstacle variant is characterized using Gaussian mixture models (GMMs). Hellinger’s distance between the GMMs is computed and used for hierarchically clustering the variants. The curriculum is determined based on the formed clusters and the average success rate in each cluster. The method was implemented on RSTAR, a highly maneuverable and reconfigurable field robot that can overcome a variety of obstacles. Learning using the automatically determined curriculum was compared to learning without a curriculum in a simulation with three obstacle types: a narrow channel, a low entrance, and a step. The results show that learning using the automatically determined curriculum enables overcoming obstacles faster and with higher success rates than learning without a curriculum for all obstacles, especially for complex obstacle variants. The developed method offers a promising method for learning motion behavior in real-word scenarios.