Temporal Self-organization for Neural Networks

Abstract

The field of artificial neural networks (ANNs) has reached a point where they are now being used in everyday products. ANNs, however, have been largely unsuccessful at processing signals that evolve over time. Temporal patterns have traditionally provided the most challenging problems for scientists and engineers and include language skills, vision skills, locomotion skills, process control, time series prediction, and many others. The fundamental concept presented in this dissertation is the formation of temporally organized neighborhoods in ANNs. This temporal self-organization enables the networks to process temporal patterns in a more organized and efficient manner. The concept is biologically inspired and uses activity diffusion to organize the processing elements of the network in an unsupervised manner. The self-organization in space and time created by my methodology has been applied to three distinct ANN architectures. The new network architectures created by adding the temporal organization are easy to implement and contain properties that are unique in the neural network field. A self-organizing map (SOM) network obtains a unique combination of long-term and short-term memory and becomes organized such that temporal patterns in the input fire sequentially ordered output PEs. These features are utilized in two different applications, a robotic landmark recognition problem and a temporally ordered vector quantization of phonemes in spoken words. When applied to the neural gas algorithm, the resulting network becomes a dynamic vector quantization network. The network anticipates the future inputs and adjusts the size of the Voronoi regions dynamically. It was used to vector quantize speech data for a digit recognition problem and to predict a chaotic signal. Lastly, the temporal organization was applied to the training of fully recurrent neural networks. It reduces the computational complexity of the training algorithm from $O(N\sp4)$ operations to only $O(N\sp2)$ operations and maintains nearly all of the power of the RTRL algorithm. This training method was tested on two inverse modeling tasks and provided a dramatic improvement in training times over the RTRL algorithm.