4
$\begingroup$

Let $d, n \in \mathbb{N}$. Moreover, let $D \subset \mathbb{R}^d$ be compact and denote with $\mathcal{C}(D, \mathbb{R}^n) $ the set of continuous functions from $D$ to $\mathbb{R}^n$. Then $\mathcal{C}(D, \mathbb{R}^n) $ is a Banach space with the usual maximum norm $\lVert f\rVert_{\infty} := \max_{x \in D} \lVert f(x) \rVert $.

Now let $\sigma : \mathbb{R} \rightarrow \mathbb{R}$ be a continuous (activation) function, $K>0$ and $\Gamma$ be a set of fixed hyperparameters for an artificial neural network such that the input layer has $d$ nodes and the output layer has $n$ nodes.

Denote with $\mathcal{H}_{\sigma, \Gamma, K} \subset \mathcal{C}(D, \mathbb{R}^n)$ the set of all neural networks with activation function $\sigma$ and fixed architecture $\Gamma$ for which all $l \in \mathbb{N}$ parameters $\theta_1,...,\theta_l$ (the biases and edgeweights) are smaller or equal than $K$ in absolute value, i.e. $\lvert \theta_i \rvert \leq K$ for all parameters $\theta_i$ of the neural network.

I want to rigorously show, that $\mathcal{H}_{\sigma, \Gamma, K}$ forms a compact topological subspace of the Banach space $(\mathcal{C}(D, \mathbb{R}^n), \lVert . \rVert_{\infty}) $.

One idea I have is to show that the mapping $j:[-K,K]^l \rightarrow \mathcal{H}_{\sigma, \Gamma, K}$ which maps a set of parameters $\Theta$ to a neural network $\Phi_{\Theta}$ is continuous. Since $[-K,K]^l$ is compact, this would imply compactness of $j([-K,K]^l) = \mathcal{H}_{\sigma, \Gamma, K}$. So far, however, I have not been successful.

I am grateful for any advice!

Kind regards and thank you very much,

Joker

$\endgroup$
2
$\begingroup$

Your result and a proof thereof can be found in Proposition 4.8 of https://arxiv.org/abs/1806.08459. The proof works exactly as you said, by showing the continuity of the "realization map".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.