Multi-head Fusion-based Actor-Critic Deep Reinforcement Learning with Memory Contextualisation for End-to-End Autonomous Navigation

Abstract

Robust autonomous navigation in dynamic and cluttered environments is a persistent challenge for mobile robotics. Conventional methods, reliant on static maps and classical algorithms, often falter when faced with frequently reconfigured obstacle layouts. Deep Reinforcement Learning (DRL) enables adaptive, map-free navigation, yet the complexity of high-resolution LiDAR data and the presence of subtle, moving obstacles complicate training and scalability. To address these issues, we present the Multi-Head Memory Contextualising TD3 (mhmcTD3) architecture, a novel DRL framework that unites multiple specialised network heads-LiDAR fusion, LiDAR Feature, Robot States, and LiDAR Memory-within a unified actor-critic backbone. This integrated design emphasises critical LiDAR states through tailored preprocessing (inversion and exponential scaling), leverages CNNs for refined feature extraction, and employs an LSTM-based LiDAR Memory head for temporal awareness. Additionally, the incorporation of the SiLU activation function and the CoRE optimiser enhances learning stability and convergence efficiency. Extensive evaluations in both simulation (using GAZEBO with ROS2) and real-world tests on a Turtlebot3 waffle_pi platform demonstrate that mhmcTD3 delivers state-of-the-art performance under diverse LiDAR resolutions. It excels at detecting and avoiding small, dynamic obstacles while maintaining adaptability in dense, fast-changing scenarios. Ablation studies further confirm the indispensable synergy among the multiple heads. Taken together, these contributions establish mhmcTD3 as a robust and generalisable DRL-based navigation solution, bridging the gap from controlled simulations to complex real-world environments.