Representation Reinforcement Learning-Based Dense Control for Point Following With State Sparse Sensing of 3-D Snake Robots

Abstract

During robot movements, the environmental states often fail to update in real-time due to interference from various factors, such as obstacle obstructions, communication disruptions, etc., which commonly results in interruptions or even failures in motion control. To achieve dense motion control under sparse state sensing, an important challenge is to predict future multiple actions based on sparse states, which is hindered by the large and complex action search space. Unfortunately, limited research has been dedicated to addressing this challenge. Therefore, this article proposes a representation reinforcement learning (RRL) based solution, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Sparse-State to Dense-Actions Latent Control</i>, designed to realize dense motion control of 3-D snake robots subject to sparse environmental state sensing, which guarantees satisfactory point following performance. In particular, by introducing a latent representation of multiple actions, the control policy optimizes latent actions to predict dense motion gaits, which significantly enhances training efficiency and motion performance. Meanwhile, to learn a compact latent variable model, three mechanisms are proposed to ensure its efficient training, semantic smoothness, and energy efficiency, facilitating exploration of the RL algorithm. To the best of our knowledge, this article provides the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">first</i> solution that enables a 3-D snake robot to successfully accomplish point following tasks under sparse state sensing. Simulation and practical experiments confirm the effectiveness, robustness, and generalizability of the proposed algorithm, with all following errors less than 0.02 m.