An intuitive explanation for neural networks as function approximators? I know we use normal linear regression for modeling functions on datasets, but can someone explain how neural networks help in approximating more complex functions, especially when they are nonlinear?
Intuitively, what does each layer adds to the whole process of approximation?
What I am looking for is an explanation of how neural networks approximate functions, and not a comparison with the biological neurons.
 A: There is some explanation for this in Duda and Hart's Pattern Recognition book. Look at section "6.2.2 Expressive power of multilayer networks". Directly quoting from there: 

It is natural to ask if every decision can be implemented by such a
  three-layer network (Eq. 6). The answer, due ultimately to Kolmogorov
  but reﬁned by others, is “yes” — any continuous function from input to
  output can be implemented in a three-layer net, given suﬃcient number
  of hidden units nH , proper nonlinearities, and weights.

...

Speciﬁcally, Kolmogorov proved that any continuous function $g(x)$
  deﬁned on the unit hypercube $I^n (I = [0, 1] \;\;\mathrm{and}\;\; n ≥ 2)$ can be
  represented in the form
  $ g(x) = \sum_{j=1}^{2n+1} \Theta_j(\sum_{i=1}^d\psi_{ij}(x_i))$ for properly chosen functions $\Theta_j$ and $\psi_{ij}$. This equation can be expressed in neural network terminology as follows: each of $2n + 1$ hidden units takes as input a sum of $d$ nonlinear functions, one for each input feature $x_i$. Each hidden unit emits a nonlinear function $\Theta$ of its total input; the
  output unit merely emits the sum of the contributions of the hidden units.

