See the Universal Approximation Theorem (arbitrary width) on wikipedia or below.
The universal approximation theorem (arbitrary width) is talking about a neural network with 1 hidden layer (input, hidden, output). In the case of a 3 layers network (1 hidden), the activation function should be evaluated twice, once on the second layer (first hidden) and once on the output layer.
Is this theorem assuming weights $v_i$ between the hidden and output layer with identity activation and no bias? If so, why do you think the authors felt no need to clarify this beyond the equation given?
It seems strange that this wouldn't be mentioned, but just thrown into the formula. I looked at the paper referenced (I found the same paper elsewhere, as their link just leads to a paper behind a paywall) on the Wikipedia article, but it seemed to be lacking this detail as well.
From Wikipedia:
"Universal approximation theorem; arbitrary width. Let $\varphi:\mathbb{R}\to\mathbb{R}$ be any continuous function (called the activation function). Let $K \subseteq \mathbb{R}^n$ be compact. The space of real-valued continuous functions on $K$ is denoted by $C(K)$. Let $\mathcal{M}$ denote the space of functions of the form
$$ F( x ) = \sum_{i=1}^{N} v_i \varphi \left( w_i^T x + b_i\right) $$
for all integers $N \in \mathbb{N}$, real constants $v_i,b_i\in\mathbb{R}$ and real vectors $w_i \in \mathbb{R}^m$ for $i=1,\ldots,N$.
Then, if and only if $\varphi$ is polynomial, the following statement is true: given any $\varepsilon>0$ and any $f\in C(K)$, there exists $F \in \mathcal{M}$ such that
$$ | F( x ) - f ( x ) | < \varepsilon $$
for all $x\in K$.
In other words, $\mathcal{M}$ is dense in $C(K)$ with respect to the uniform norm if and only if$\varphi$ is nonpolynomial.
This theorem extends straightforwardly to networks with any fixed number of hidden layers: the theorem implies that the first layer can approximate any desired function, and that later layers can approximate the identity function. Thus any fixed-depth network may approximate any continuous function, and this version of the theorem applies to networks with bounded depth and arbitrary width."