In Wikipedia's Statement of The Universal Approximation Theorem, is it taking identity activation on the output layer with no bias?

Question

See the Universal Approximation Theorem (arbitrary width) on wikipedia or below.

The universal approximation theorem (arbitrary width) is talking about a neural network with 1 hidden layer (input, hidden, output). In the case of a 3 layers network (1 hidden), the activation function should be evaluated twice, once on the second layer (first hidden) and once on the output layer.

Is this theorem assuming weights $v_i$ between the hidden and output layer with identity activation and no bias? If so, why do you think the authors felt no need to clarify this beyond the equation given?

It seems strange that this wouldn't be mentioned, but just thrown into the formula. I looked at the paper referenced (I found the same paper elsewhere, as their link just leads to a paper behind a paywall) on the Wikipedia article, but it seemed to be lacking this detail as well.

---

From Wikipedia:

"Universal approximation theorem; arbitrary width. Let $\varphi:\mathbb{R}\to\mathbb{R}$ be any continuous function (called the activation function). Let $K \subseteq \mathbb{R}^n$ be compact. The space of real-valued continuous functions on $K$ is denoted by $C(K)$. Let $\mathcal{M}$ denote the space of functions of the form

$$ F( x ) = \sum_{i=1}^{N} v_i \varphi \left( w_i^T x + b_i\right) $$

for all integers $N \in \mathbb{N}$, real constants $v_i,b_i\in\mathbb{R}$ and real vectors $w_i \in \mathbb{R}^m$ for $i=1,\ldots,N$.

Then, if and only if $\varphi$ is polynomial, the following statement is true: given any $\varepsilon>0$ and any $f\in C(K)$, there exists $F \in \mathcal{M}$ such that

$$ | F( x ) - f ( x ) | < \varepsilon $$

for all $x\in K$.

In other words, $\mathcal{M}$ is dense in $C(K)$ with respect to the uniform norm if and only if$\varphi$ is nonpolynomial.

This theorem extends straightforwardly to networks with any fixed number of hidden layers: the theorem implies that the first layer can approximate any desired function, and that later layers can approximate the identity function. Thus any fixed-depth network may approximate any continuous function, and this version of the theorem applies to networks with bounded depth and arbitrary width."

Answer 1

Most common activation functions, when applied at the final layer, will impose a restriction on the range of the function represented by the network. For example, if using a ReLU activation on the final layer, then the output of the network will lie in $\{x\in\mathbb{R}^n: x_i\ge 0, \forall i\}$. Similarly, if using a sigmoid on the final layer, then the output will lie in $\{x\in\mathbb{R}^n: |x_i|\leq 1, \forall i\}$. If the output of the network is constrained to a proper subset of $\mathbb{R}^n$, then the network cannot be a universal function approximator, so that is likely why the authors did not explicitly mention this.

Answer 2

See the last paragraph of A closer look at the approximation capabilities of neural networks § 2.1 Notation and Definitions. With regard to the map

$$x \mapsto w_{0,j}^{(2)} + \sum_{i=1}^N w_{i,j}^{(2)} \sigma(\mathbf{w}_i^{(1)} \cdot (1,x))$$

it says:

Although some authors... do not explicitly include bias weights in the output layer, the reader should check that if $\sigma$ is not identically zero, say $\sigma(y_0) \neq 0$, then having a bias weight $w_{0,j}^{(2)} = c$ is equivalent to setting $w_{0,j}^{(2)} = 0$ (i.e. no bias weight in the output layer) and introducing an $(N+1)$-th hidden unit, with corresponding weights $w_{0,N+1}^{(1)} = y_0, w_{i,N+1}^{(1)} = 0$ for all $1 \leq i \leq n$, and $w_{N+1,j}^{(2)} = \frac{c}{\sigma(y_0)}$; this means our results also apply to neural networks without bias weights in the output layer (but with one additional hidden unit).