Neural Network topology

Question

I am reading about Neural Networks and I always find the same topology for the underlying network: an input layer followed by some hidden layer and then an output layer. Why are not proposed other topologies for the network connecting neurons? Is the standard topology a sort of a "well behaved" topology for which we can apply our learning algortihms and we are sure of convergence?

Answer 1

There are many other topologies. What you are describing is the basic feed-forward neural network. The feedforward topology cannot be more complex than input->hidden->output. Feed forward means that the inputs to one layer depend only on the outputs from another (or, in the case of the input layer itself, they depend on whatever the inputs to the network are). what's missing in the FF topology is that it is possible to create a Neural network in which the inputs to some of the previous layers are the outputs of future layers.

These networks are extremely cool, but there are so many ways to to create them that you often don't see their topologies described in introductory stuff. The big benefit of such a network is that the network can re-process old data. This lets you do things like search for time-dependent or transient events without providing a huge vector of inputs that represents the time series of the quantity under consideration.

Perhaps the problem is that there is no such thing as a "prototypical artificial neural network". We often begin learning about them by learning about the basic perceptron and its generalizations, but then you need to spend a lot of time studying individual network types before you understand their unique uses, advantages, and disadvantages.

To get started on learning about convolutional neural network and other more complicated structures, Wikipedia is a good resource

Answer 2

You might find this paper, titled "How many hidden layers and nodes" very relevant as it addresses your question directly. http://dstath.users.uth.gr/papers/IJRS2009_Stathakis.pdf

In general,

The basic topology you describe-three layers, one hidden-is sufficient to approximate any continuous function on compact subsets of $\mathbb{R^n}$, a result which is due to Cybenko.

Cybenko opted for simplicity and generality and this architecture achives this. Adding more layers or choosing to connect neurons in a different fashion generally means better approximation but you have to take into account the particularities of the problem, the degree of approximation you wish to achieve, the computational power available. To a significant degree it's trial-and-error and experience.

On a sidenote, besides Cybenko's approximation, there is another- far stronger-result, Kolmogorov's Superposition Theorem (Or Kolmogorov-Arnold Representation) which states how a continuous multivariate function can be exactly represented. Hecht-Nielsen later showed how this can help in designing a neural network.