# Are neural networks with bounded parameters a compact subset of the Banach space of continuous functions?

Let $$d, n \in \mathbb{N}$$. Moreover, let $$D \subset \mathbb{R}^d$$ be compact and denote with $$\mathcal{C}(D, \mathbb{R}^n)$$ the set of continuous functions from $$D$$ to $$\mathbb{R}^n$$. Then $$\mathcal{C}(D, \mathbb{R}^n)$$ is a Banach space with the usual maximum norm $$\lVert f\rVert_{\infty} := \max_{x \in D} \lVert f(x) \rVert$$.

Now let $$\sigma : \mathbb{R} \rightarrow \mathbb{R}$$ be a continuous (activation) function, $$K>0$$ and $$\Gamma$$ be a set of fixed hyperparameters for an artificial neural network such that the input layer has $$d$$ nodes and the output layer has $$n$$ nodes.

Denote with $$\mathcal{H}_{\sigma, \Gamma, K} \subset \mathcal{C}(D, \mathbb{R}^n)$$ the set of all neural networks with activation function $$\sigma$$ and fixed architecture $$\Gamma$$ for which all $$l \in \mathbb{N}$$ parameters $$\theta_1,...,\theta_l$$ (the biases and edgeweights) are smaller or equal than $$K$$ in absolute value, i.e. $$\lvert \theta_i \rvert \leq K$$ for all parameters $$\theta_i$$ of the neural network.

I want to rigorously show, that $$\mathcal{H}_{\sigma, \Gamma, K}$$ forms a compact topological subspace of the Banach space $$(\mathcal{C}(D, \mathbb{R}^n), \lVert . \rVert_{\infty})$$.

One idea I have is to show that the mapping $$j:[-K,K]^l \rightarrow \mathcal{H}_{\sigma, \Gamma, K}$$ which maps a set of parameters $$\Theta$$ to a neural network $$\Phi_{\Theta}$$ is continuous. Since $$[-K,K]^l$$ is compact, this would imply compactness of $$j([-K,K]^l) = \mathcal{H}_{\sigma, \Gamma, K}$$. So far, however, I have not been successful.

I am grateful for any advice!

Kind regards and thank you very much,

Joker