4
$\begingroup$

Let $d, n \in \mathbb{N}$. Moreover, let $D \subset \mathbb{R}^d$ be compact and denote with $\mathcal{C}(D, \mathbb{R}^n) $ the set of continuous functions from $D$ to $\mathbb{R}^n$. Then $\mathcal{C}(D, \mathbb{R}^n) $ is a Banach space with the usual maximum norm $\lVert f\rVert_{\infty} := \max_{x \in D} \lVert f(x) \rVert $.

Now let $\sigma : \mathbb{R} \rightarrow \mathbb{R}$ be a continuous (activation) function, $K>0$ and $\Gamma$ be a set of fixed hyperparameters for an artificial neural network such that the input layer has $d$ nodes and the output layer has $n$ nodes.

Denote with $\mathcal{H}_{\sigma, \Gamma, K} \subset \mathcal{C}(D, \mathbb{R}^n)$ the set of all neural networks with activation function $\sigma$ and fixed architecture $\Gamma$ for which all $l \in \mathbb{N}$ parameters $\theta_1,...,\theta_l$ (the biases and edgeweights) are smaller or equal than $K$ in absolute value, i.e. $\lvert \theta_i \rvert \leq K$ for all parameters $\theta_i$ of the neural network.

I want to rigorously show, that $\mathcal{H}_{\sigma, \Gamma, K}$ forms a compact topological subspace of the Banach space $(\mathcal{C}(D, \mathbb{R}^n), \lVert . \rVert_{\infty}) $.

One idea I have is to show that the mapping $j:[-K,K]^l \rightarrow \mathcal{H}_{\sigma, \Gamma, K}$ which maps a set of parameters $\Theta$ to a neural network $\Phi_{\Theta}$ is continuous. Since $[-K,K]^l$ is compact, this would imply compactness of $j([-K,K]^l) = \mathcal{H}_{\sigma, \Gamma, K}$. So far, however, I have not been successful.

I am grateful for any advice!

Kind regards and thank you very much,

Joker

$\endgroup$

1 Answer 1

2
$\begingroup$

Your result and a proof thereof can be found in Proposition 4.8 of https://arxiv.org/abs/1806.08459. The proof works exactly as you said, by showing the continuity of the "realization map".

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .