Neighbor-Based Decentralized Training Strategies for Multi-Agent Reinforcement Learning

Abstract

Multi-agent deep reinforcement learning has demonstrated significant potential as a promising framework for developing autonomous agents capable of operating within complex, multi-agent environments in a wide range of domains, like robotics, traffic management, and video games. Centralized training with decentralized execution has emerged as the predominant training paradigm, demonstrating significant effectiveness in learning complex policies. However, its reliance on a centralized learner necessitates that agents acquire policies offline and subsequently execute them online. This constraint motivates the exploration of decentralized training methodologies. Despite their greater flexibility, decentralized approaches often face critical challenges, like: slower convergence rates, higher instability and lower performance compared to centralized methods. Therefore, this paper proposes three neighbor-based decentralized training strategies based on the Deep-Q Learning algorithm and investigates their effectiveness as a viable alternative to centralized training. We evaluate experience sharing, k-nearest neighbor averaging, and k-nearest neighbor consensus methods in a cooperative multi-agent environment and compare their performance against centralized training and totally decentralized training. Our results show that neighbor-based methods can achieve comparable performance to centralized training while offering improved scalability and communication efficiency.