Enhancing collaboration in uncertain environment: Multi-Agent Reinforcement Learning for underwater monitoring

Abstract

Underwater monitoring is extremely complex due to the lack of a global localization system, limited communication and environmental factors such as turbidity and darkness that limit visibility, affecting control and situational awareness. Typically, monitoring relies on a single autonomous underwater vehicle (AUV) or a set of independent AUVs; techniques which are prone to failure as they rely only on onboard odometry and sensors, making missions vulnerable to malfunctions, damage, and noise. To address these challenges, we propose a Multi-Agent Reinforcement Learning (MARL) framework to enable cooperation among multiple AUVs, mitigating the limitations of the underwater environment. Our in-silico solution focuses on a group of robots learning a strategy to follow a partially hidden underwater pipe without global localization, while dealing with environmental disturbances affecting sensors and actuators. The numerosity of the agents, and most importantly their collaboration, helps overcome underwater visibility constraints. By sharing relative position information of neighboring agents with respect to the pipe, navigation is improved. By introducing quantitative measures for pipe exploration, we show that cooperation significantly enhances system performance compared to independent agents. Emerging collaboration among robots allows the swarm to complete pipe inspections faster and more efficiently than non-cooperative baseline models of non-interacting agents, even under extremely reduced visibility scenarios. Moreover, single agents also benefit from cooperation, learning effective policies more quickly and covering a longer portion of the pipe. Finally, our model guarantees explainability. We analyze learned strategies and provide a visualization method that allows the interpretation of the learned policies. • Reinforcement learning is applied to pipeline following by underwater robotic agent. • In conditions of poor visibility, a single agent is not able to complete the mission. • In contrast, multi-agent team completes this task using reinforcement learning. • Swarm collaboration enables faster and more efficient task completion. • Collaboration enables agents develop more efficient individual strategies.