Encoding robotic sensor states for Q-learning using the self-organizing map

Abstract

The self-organizing map (SOM) [1] reduces a large input space to a fixed-size output space. It is trained by means of an unsupervised learning algorithm. One application of the SOM is to transform a set of robot sensor readings into a state space suitable as the input for the reinforcement learning algorithm Q-learning [5]. We have implemented two different formulations of this concept [2][3] using the Lego Mindstorms NXT robot [8], a robot commonly used in undergraduate computer science courses. We compared the performance of our implementations against a traditional Q-learning implementation. We found that the number and type of sensors encoded by the SOM has a significant impact on the quality of the behavior the robot learns.