I was looking at the differences between these mathematical theorems:
Universal Approximation Theorem (https://en.wikipedia.org/wiki/Universal_approximation_theorem)
Kolmogorov Representation Theorem (https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_representation_theorem)
Both of these theorems concern themselves with function approximating other functions through compositions.
In the case of the Universal Approximation Theorem, it has been shown that sums of function compositions (in the form of a Neural Network) have the ability to approximate any function to any level of error. The Universal Approximation Theorem is said to be the main mathematical theorem that justifies the success of Neural Networks. Mathematically, Neural Networks can be interpreted as "affine compositions of functions".
In the case of the Kolmogorov Representation Theorem, it has been shown that any function can be fully represented (i.e. no error) also through finite sums of compositions. The Kolmogorov Representation Theorem has a "similar" form as the Universal Approximation Theorem - although they are both different.
If I have understood all this correctly - both of these theorems make claims about the ability to approximate any function using compositions of functions - however, one of these theorems is specifically called the "Universal Approximation Theorem".
If I have understood all this correctly - it seems to me that the Kolmogorov's Representation Theorem only guarantees the existence of such a function, yet provides us no meaningful method to "recover" this function ... whereas the Universal Approximation Theorem is effectively making a similar claim about an already existing method (i.e. neural networks) of recovering such a function.
- Is my understanding of the above correct?
- Have I correctly identified the differences between the Universal Approximation Theorem and Kolomogorov's Representation Theorem?
- Why are neural networks based on the "Universal Approximation Theorem" and not the "Kolmogorov Representation Theorem"?