Recently I read a paper on deep neural networks which explained that the ReLU activation function $f(x) = max(0,x)$ which is piece-wise linear is highly-effective within large networks because input data that is projected into high dimensional spaces allows linear representations of a nonlinear function. However, they don't mention the theorem they use.

After more reflection I think this makes sense because any real-valued continuous function is linear in the infinite dimensional space consisting of continuous functions. Having said this, I don't remember being taught a theorem from which such conclusions may be drawn.

Might someone know the standard name for this theorem? I'm very interested in generalisations of this theorem.

X. Glorot et al. Deep Sparse Rectifier Neural Networks. 2011.


1 Answer 1


Is Universal Approximation Theorem what you're looking for?

  • $\begingroup$ No, although the capacity of neural networks as complex function approximators might be a special consequence of the result I'm looking for. $\endgroup$
    – user93511
    Jun 2, 2017 at 23:21

You must log in to answer this question.