Is XCS Suitable For Problems with Temporal Rewards?

Abstract

XCS [1], the accuracy-based classifier system, provides a very brilliant way to merge genetic algorithmic (GA) rule learning and reinforcement learning (RL) methodologies together. This makes it suitable for a wide range of applications where generalisation over decision making states is desirable. Also, its Q-learning-oriented prediction update scheme enables it to handle multi-step problems adequately. This paper reports how the intertwined spirals problem, initially a popular benchmark in classification, was modified by the authors to verify XCS’s suitability for behavioural design of robotic systems. When the results obtained were not as expected, investigations were continued until a rather surprising conclusion was drawn: XCS cannot handle very simple problems if the rewards are temporally-oriented, even if the reward is extremely short-delayed.