Optimal Network Size

The size of a neural network is characterized by the number of hidden neurons, the network complexity by the number of hidden neurons and by their interrelationship, e.g. by the number of hidden layers in a multi layer perceptron architecture. If the network is too small, it will not completely solve a particular problem or parts of it, and a too complex network has the tendency to memorize the data. It concentrates too much on the data presented for learning and tends toward modelling the noise. These cases are called under fitting or over fitting respectively.


Over Sized Model		Regular Model

The selection of an optimal network size is still a big issue since there is up to now no analytical solution to that problem available. One acceptable approach to that problem is a heuristic one, training several networks with increasing complexity using a learning subset of the training data and concurrently observing the error of a validation subset. After training is completed, the network with the lowest validation error is the preferred one.

Using multi layer perceptron architecture (MLP or GMP) that approach has two dimensions, each of them covering an infinite number of possibilities yielding into two major questions.

To answer these questions is a big issue since there are infinite times infinite combinatorial possibilities to that approach and until now no analytical solution is available. To respond nevertheless to that issue we migrate off from thinking in layers to thinking in blocks.

Between input and output layer are no more an arbitrary and, as a matter of fact, unknown number of hidden layers, but only one hidden block. That block covers the hidden neurons in a manner that all of them are forward connected in all combinatorial possibilities. We call that kind of network architecture a completely connected perceptron (CCP).

The question still remaining is how many hidden neurons within the hidden block? In combination with an automatic network growing algorithm the approach to the optimal network size accomplishes in a one dimensional heuristic way.

The training process starts with an arbitrary number of hidden neurons, adding one hidden neuron after one and retraining each enlarged network again. This process is repeated until a proper solution is obtained. For the selection of the optimal network size a separate data partition, called the validation data subset, is used. A testing subset may be used in addition for QC.

cVision supports multi layer perceptron architectures with and without shortcut connections, both with arbitrary number of hidden layer and hidden neurons and of course the completely connected perceptron. Growing is a feature available for all available architectures.