Turbo-IRL: Enhancing multi-agent systems using turbo decoding-inspired deep maximum entropy inverse reinforcement learning

摘要

• We propose a novel parallel framework for multi-agent IRL with a shared objective. • The architecture of Turbo-IRL is inspired by the iterative nature of turbo decoding. • Turbo-IRL uses a common reward, helping agents refine their individual rewards. • Our proposed method (Turbo-IRL) converges faster than Deep IRL in various scenarios. • In unbalanced settings, under-trained agents gain from others’ data. Inverse Reinforcement Learning (IRL) involves analyzing expert agents’ behavior to uncover the rationality behind their actions, with applications in robotics, autonomous systems, gaming, and the animation industry. Conventional Multi-Agent IRL (MA-IRL) approaches typically assume either fully cooperative or fully independent agent objectives. However, real-world scenarios often involve partially aligned goals comprising both shared and agent-specific components. To address this complexity, we propose Turbo-IRL, a novel, parallelizable MA-IRL framework that extends Deep Maximum Entropy IRL with an iterative information exchange mechanism inspired by Turbo Decoding (TD) in coding theory. Turbo-IRL consists of multiple agent-specific IRL modules, each leveraging deep neural networks to recover complex reward structures. Through iterative communication, agents refine their estimates of shared rewards while concurrently discovering individualized reward components. This information exchange mechanism allows them to benefit from others’ trajectories, which convey partial information about their common goals. This is particularly advantageous for under-trained agents with a small number of trajectories in unbalanced scenarios. Our simulations across multiple scenarios with different levels of overlap and similarity between agents’ reward functions show a considerable gain for the proposed Turbo-IRL framework compared to the benchmark Deep IRL method. Specifically, we achieve about 50% and 44% improvement in terms of MSE in recovering the true reward function for similarity levels 75% and 25%, utilizing the same number of total trajectories. Conversely, our method achieves equivalent performance utilizing much fewer (about 44% for a 3-agent system). Furthermore, on the standard MPE simple_spread benchmark, Turbo-IRL achieves the lowest MAE of 1.04 in episode return, outperforming several state-of-the-art baselines, including MA-GAIL, MA-AIRL, Behavior Cloning, and MIFQ (38% to 87% improvement) – underscoring its efficacy, scalability, and generalization capability in complex multi-agent environments with partially shared objectives.