Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

We use normal linear regression for modelling functions on datasets . But Can someone explain how neural networks help in approximating more complex ,especially non-linear functions ? intuitively , like what each layer adds to the whole process of approximation etc . Also , when does using artificial networks more prefferable ? What i'am looking for is an explanation of neural network as function approximators , and not a comparison with the biological neurons .


share|improve this question

1 Answer 1

up vote 1 down vote accepted

There is some explanation for this in Duda and Hart's Pattern Recognition book. Look at section "6.2.2 Expressive power of multilayer networks". Directly quoting from there:

It is natural to ask if every decision can be implemented by such a three-layer network (Eq. 6). The answer, due ultimately to Kolmogorov but refined by others, is “yes” — any continuous function from input to output can be implemented in a three-layer net, given sufficient number of hidden units nH , proper nonlinearities, and weights.


Specifically, Kolmogorov proved that any continuous function $g(x)$ defined on the unit hypercube $I^n (I = [0, 1] \;\;\mathrm{and}\;\; n ≥ 2)$ can be represented in the form $ g(x) = \sum_{j=1}^{2n+1} \Theta_j(\sum_{i=1}^d\psi_{ij}(x_i))$ for properly chosen functions $\Theta_j$ and $\psi_{ij}$. This equation can be expressed in neural network terminology as follows: each of $2n + 1$ hidden units takes as input a sum of $d$ nonlinear functions, one for each input feature $x_i$. Each hidden unit emits a nonlinear function $\Theta$ of its total input; the output unit merely emits the sum of the contributions of the hidden units.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.