Universal approximation theorem states that "the standard multilayer feed-forward network with a single hidden layer, which contains finite number of hidden neurons, is a universal approximator among continuous functions on compact subsets of $R^n$, under mild assumptions on the activation function."

I understand what this means, but the relevant papers are too far over my level of math understanding to grasp why it is true or how a hidden layer approximates non-linear functions.

So, in terms little more advanced than basic calculus and linear algebra, how does a feed-forward network with one hidden layer approximate non-linear functions? The answer need not necessarily be totally concrete.

I also posted this question at TCS, and CV. Previously, no one had given a solution. But now, here is a really excellent and comprehensive answer.

  • $\begingroup$ maybe cstheory.stackexchange.com or cs.stackexchange.com might be better suited for this question. $\endgroup$
    – robjohn
    May 7, 2013 at 22:02
  • $\begingroup$ @robjohn I was also wondering if cstheory might be a better fit. What would be the best way to approach that possibility? I assume the correct procedure would be to migrate the question, but I'm a bit wary of possibly migrating to the wrong place. $\endgroup$ May 7, 2013 at 22:14
  • $\begingroup$ Just repost this on cstheory. You can either delete the question here or leave it. If you leave it here, link it to the question on cstheory, and link to here from the question on cstheory. $\endgroup$
    – robjohn
    May 7, 2013 at 22:18
  • $\begingroup$ @robjohn cool, thanks. $\endgroup$ May 7, 2013 at 22:19

1 Answer 1


If the hidden units are radial basis functions (i.e., they have a peak response when the input pattern is close in a Euclidean distance sense to the parameter vector of the hidden unit), then each hidden unit basically generates a "bump". A superposition of such "bumps" can then be used to approximate an arbitrary function. Other types of hidden units such as "sigmoidal" units also have this type of "bump" response property.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .