Noise Resilience of Successor and Predecessor Feature Algorithms in One- and Two-Dimensional Environments

摘要

Noisy inputs pose significant challenges for reinforcement learning (RL) agents navigating real-world environments. While animals demonstrate robust spatial learning under dynamic conditions, the mechanisms underlying this resilience remain understudied in RL frameworks. This paper introduces a novel comparative analysis of predecessor feature (PF) and successor feature (SF) algorithms under controlled noise conditions, revealing several insights. Our key innovation lies in demonstrating that SF algorithms achieve superior noise resilience compared to traditional approaches, with cumulative rewards of 2216.88±3.83 (mean ± SEM), even under high noise conditions (σ=0.5) in one-dimensional environments, while Q learning achieves only 19.22±0.57. In two-dimensional environments, we discover an unprecedented nonlinear relationship between noise level and algorithm performance, with SF showing optimal performance at moderate noise levels (σ=0.25), achieving cumulative rewards of 2886.03±1.63 compared to 2798.16±3.54 for Q learning. The λ parameter in PF learning is a significant factor, with λ=0.7 consistently achieving higher λ values under most noise conditions. These findings bridge computational neuroscience and RL, offering practical insights for developing noise-resistant learning systems. Our results have direct applications in robotics, autonomous navigation, and sensor-based AI systems, particularly in environments with inherent observational uncertainty.