Let $d, n \in \mathbb{N}$. Moreover, let $D \subset \mathbb{R}^d$ be compact and denote with $\mathcal{C}(D, \mathbb{R}^n) $ the set of continuous functions from $D$ to $\mathbb{R}^n$. Then $\mathcal{C}(D, \mathbb{R}^n) $ is a Banach space with the usual maximum norm $\lVert f\rVert_{\infty} := \max_{x \in D} \lVert f(x) \rVert $.
Now let $\sigma : \mathbb{R} \rightarrow \mathbb{R}$ be a continuous (activation) function, $K>0$ and $\Gamma$ be a set of fixed hyperparameters for an artificial neural network such that the input layer has $d$ nodes and the output layer has $n$ nodes.
Denote with $\mathcal{H}_{\sigma, \Gamma, K} \subset \mathcal{C}(D, \mathbb{R}^n)$ the set of all neural networks with activation function $\sigma$ and fixed architecture $\Gamma$ for which all $l \in \mathbb{N}$ parameters $\theta_1,...,\theta_l$ (the biases and edgeweights) are smaller or equal than $K$ in absolute value, i.e. $\lvert \theta_i \rvert \leq K$ for all parameters $\theta_i$ of the neural network.
I want to rigorously show, that $\mathcal{H}_{\sigma, \Gamma, K}$ forms a compact topological subspace of the Banach space $(\mathcal{C}(D, \mathbb{R}^n), \lVert . \rVert_{\infty}) $.
One idea I have is to show that the mapping $j:[-K,K]^l \rightarrow \mathcal{H}_{\sigma, \Gamma, K}$ which maps a set of parameters $\Theta$ to a neural network $\Phi_{\Theta}$ is continuous. Since $[-K,K]^l$ is compact, this would imply compactness of $j([-K,K]^l) = \mathcal{H}_{\sigma, \Gamma, K}$. So far, however, I have not been successful.
I am grateful for any advice!
Kind regards and thank you very much,
Joker
