Recently I read a paper on deep neural networks which explained that the ReLU activation function $f(x) = max(0,x)$ which is piece-wise linear is highly-effective within large networks because input data that is projected into high dimensional spaces allows linear representations of a nonlinear function. However, they don't mention the theorem they use.
After more reflection I think this makes sense because any real-valued continuous function is linear in the infinite dimensional space consisting of continuous functions. Having said this, I don't remember being taught a theorem from which such conclusions may be drawn.
Might someone know the standard name for this theorem? I'm very interested in generalisations of this theorem.
X. Glorot et al. Deep Sparse Rectifier Neural Networks. 2011.
