I was reading about the mathematical background behind Neural Networks to try and understand why a Neural Network is in theory able to "work". I came across the following link: https://en.wikipedia.org/wiki/Universal_approximation_theorem
If I understand correctly, the original Universal Approximation Theory of Neural Networks was used to first show that a Neural Network with 2 Layers and "Arbitrary Depth" (e.g. an infinite number of neurons) is able to approximate any function to an arbitrary level of precision.
However, the "issue" with this theorem was that a Neural Network with an "arbitrarily large number of neurons" might be able to perfectly approximate any function - but a Neural Network with an "arbitrarily large number of neurons" will be computationally impossible (i.e. "too deep"). Thus, extensive research was done to extend the "fruits" of the Universal Approximation Theorem (i.e. ability of Neural Networks to approximate any function arbitrarily well) to Neural Networks of "non-arbitrary" (i.e. bounded) depth. Thus, how well can Neural Networks perform that have "many layers (instead of only two layers) but a non-arbitrary number of neurons (i.e. non-arbitrary depth)"?
I was reading the abstracts of these two papers:
I copy and pasted the abstracts below:
Abstract 1: The standard Universal Approximation Theorem for operator neural networks (NNs) holds for arbitrary width and bounded depth. Here, we prove that operator NNs of bounded width and arbitrary depth are universal approximators for continuous nonlinear operators. In our main result, we prove that for non-polynomial activation functions that are continuously differentiable at a point with a nonzero derivative, one can construct an operator NN of width five, whose inputs are real numbers with finite decimal representations, that is arbitrarily close to any given continuous nonlinear operator.
Abstract 2: The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonaffine continuous function, with a continuous nonzero derivative at some point.
Based on the descriptions provided in these two abstracts - I think I am incorrectly understanding the Universal Approximation Theorem or I am incorrectly understanding the definitions of the words "bounded" and "arbitrary" . In both abstracts, it is mentioned that "the standard Universal Approximation Theorem for operator neural networks (NNs) holds for arbitrary width and bounded depth".
However, I thought it was precisely the opposite - the classic/original/standard Universal Approximation Theorem was intended for "bounded width and arbitrary depth" (e.g. 2 layers and infinite Neurons), and significant research was done to extend the results of this theorem to prove the existence of a class of Neural Networks with similar approximation capabilities having "arbitrary width and bounded depth".
Isn't this exactly the opposite?
- Can someone please help in clarifying the confusion I am having?
PS: I have always been interested in understanding the mathematical definition of the term "Dense" :
Based on the definition of "Dense" provided here (https://en.wikipedia.org/wiki/Dense_set), the above statement seems to be suggesting that ALL Neural Networks belonging to "N" are "arbitrarily close" to ALL D-dimensional functions that map X to Y (i.e. the set "C").
In other words, does this mean that suppose you have the point "A1" belonging to some function "F1" (and the function "F1" is in the set "C") - in theory, a Neural Network belonging to the class "N" would be able to provide an approximation to the point "A1" (e.g. call this approximation as "A2") : And "A2" will be arbitrarily close to "A1"?
And of course, this statement regarding quality of approximation could be extended to all elements belonging to the same set as "A1" and all elements belonging to the same set as "A2"?