Cooperative multi-agent reinforcement learning for robotic systems: A review

Abstract

This review provides a comprehensive analysis of cooperative Multi-Agent Reinforcement Learning (MARL) approaches for robotic systems, with particular emphasis on methodological foundations, practical implementations, and emerging challenges. We first examine the evolution of distributed intelligence in robotics, tracing its development from early architectures to modern learning-based frameworks. Our analysis focuses on two complementary paradigms: cooperative methods utilizing Centralized Training with Decentralized Execution (CTDE), and hierarchical approaches that address complexity through temporal and task decomposition. We systematically compare actor-critic methods, value-based approaches, and hierarchical frameworks across theoretical foundations, implementation characteristics, and application domains spanning aerial, ground, and maritime robotics. Our comparative analysis reveals important trade-offs between expressiveness, computational efficiency, and implementation complexity, highlighting that method selection must align with specific application requirements. Furthermore, we identify critical challenges, including the sim-to-real gap, scalability constraints, communication limitations, safety verification, and coordination in heterogeneous teams, mapping promising research directions to address these barriers to widespread deployment. This survey bridges theoretical understanding with practical implementation, providing a structured framework for researchers and practitioners working on multi-agent learning for advanced robotic systems.