Local Adaptive Learning Rules

Learning means updating the synaptic weights of a neural network in a way that the actual network output approaches as close as possible to the desired output. Many different and highly sophisticated learning rules are developed until now, they may be partitioned into three major groups, global learning, local learning and local adaptive learning. All of these require a learning rate that states how much a weight will change with respect to the local error gradient. If that gradient is small then the step is small too, compared to a large gradient with identical learning rate.

So it is hard to accept that a single learning rate, as applied in global learning, is able to fulfil the requirements of all the (hundreds or thousands) of synaptic weights, each of them encompassing most likely different error gradients. Using local learning rules is more sophisticated since each weight has its private learning rate, but nevertheless the training process is jeopardized by improper selection of these learning rates. Undulations in steep error valleys and very slow convergence on flat error gradients can slow down the network training process dramatically.

Local adaptive learning rules have, as in conventional local learning, no single learning rate responsible for the particular weights. Each weight has of course its private learning rate, but, and that's the crucial feature, each of these learning rates are continuously optimized during the whole training process yielding an extremely fast learning speed. Throughout the integration of local adaptive learning rules the question for an optimal learning rate is no longer a major issue for a successful training of neural networks.

Local adaptive learning is the default in cVision, nevertheless cVision supports local learning as well as global learning for comparison reasons.