Addressing imperfect symmetry: A novel symmetry-learning actor-critic extension

Abstract

Symmetry, a fundamental concept to understand our environment, often oversimplifies reality from a mathematical perspective. Humans are a prime example, deviating from perfect symmetry in terms of appearance and cognitive biases (e.g. having a dominant hand). Nevertheless, our brain can easily overcome these imperfections and efficiently adapt to symmetrical tasks. The driving motivation behind this work lies in capturing this ability through reinforcement learning. To this end, we introduce Adaptive Symmetry Learning (ASL) — a model-minimization actor-critic extension that addresses incomplete or inexact symmetry descriptions by adapting itself during the learning process. ASL consists of a symmetry fitting component and a modular loss function that enforces a common symmetric relation across all states while adapting to the learned policy. The performance of ASL is compared to existing symmetry-enhanced methods in a case study involving a four-legged ant model for multidirectional locomotion tasks. The results show that ASL can recover from large perturbations and generalize knowledge to hidden symmetric states. It achieves comparable or better performance than alternative methods in most scenarios, making it a valuable approach for leveraging model symmetry while compensating for inherent perturbations. • We introduce Adaptive Symmetry Learning (ASL), a model-minimization actor-critic extension that is able to handle incomplete or inexact symmetry descriptions by adapting itself during the learning process. • ASL is composed of two parts — symmetry fitting, described above; and a new loss function, capable of applying the learned descriptions based on their importance, while actively avoiding neutral states and disadvantageous updates. • We propose modifications to two existing symmetry loss functions — MSL (Yu et al., 2018) and PSL (Kasaei et al. 2021) — extending them with value losses and the capacity to handle involutory transformations, such as most rotations. • We present the case study of an ant robot, where we compare the performance of MSL, PSL, ASL, and vanilla PPO (Schulman et al., 2017) in a set of ant locomotion scenarios. We test controlled and realistic symmetry perturbations, as well as partial goal observation, where the policy must discover the best symmetric gait without exploring the symmetric side, while acknowledging the existence of perturbations.