Extending cortical-basal inspired reinforcement learning model with success-failure experience

摘要

Neurocognitive studies showed that neurons of the orbitofrontal cortex get activated for expectation of immediate reward. Therefore they are the key reward structure in the brain. It was also shown that neurons in the anterior cingulate cortex work as an early warning system that prevents repeating mistakes. This paper introduces an extended model of reinforcement learning in the cortex-basal ganglia network by the hypothetical involvement of two cortical regions, the orbitofrontal cortex and the anterior cingulate cortex. In order to prove the effectiveness of the approach, we propose an enhanced actor-critic method that is guided by experiences of success and failure. Failures help the agent to explore regions by avoiding past mistakes. Successful experiences allow to exploit those regions that guarantee the agent to reach its goal. First, the method was applied to a 2-D grid problem, where an agent had to reach its goal by avoiding obstacles in its path. Second, the proposed RL model was used to optimize the learning policy of how to play bowling by the NAO humanoid robot. The results showed significant improvement using the enhanced actor-critic method both in terms of performance and rate of learning compared with the standard actor-critic method.