I need a definition of neural networks in terms of mathematical mapping syntax. Since neural networks come in all different shapes I find it a little hard to come up with a definition that comprises all different types. For the case of feed forward networks without cycles, loops and skip connections a solid definition could be
$ \mathbb{R}^{n_0} \ni x_0 \mapsto x_N \in \mathbb{R}^{n_N}$ where $x_{k+1} = F_k(x_k)$ and $F_k: \mathbb{R}^{n_k} \to \mathbb{R}^{n_{k+1}} $.
In the case where forward skip connections are admissable, I think the definition would change to
$ \mathbb{R}^{n_0} \ni x_0 \mapsto x_N \in \mathbb{R}^{n_N}$ where $x_{k+1} = F_k(x_0,x_1,...,x_k)$ and $F_k: \mathbb{R}^{\sum n_k} \to \mathbb{R}^{\sum n_k + n_{k+1}} $
In the following paper "Neural Ordinary Differential Equations" the authors state
Models such as residual networks, recurrent neural network decoders, and normalizing flows build complicated transformations by composing a sequence of transformations to a hidden state: $h_{t+1} = h_t + f(h_t,\theta_t)$
This can obviously only be true for networks which same dimension in each layer.
Also other papers that I read claimed that residual networks (i.e. networks containing skip connections) can be represented in the above way. My questions are
- How can it be that recurrent neural network (i.e. networks contaning loops and/or cycles) can be represented in the way from the block quote ? Where happens the loop in the formula ?
- Also, how can residual networks with skip connections between non-consecutive layers be written in that way ? Where do we see the skip connections in the formula ?
- How would I need to change the second definition I gave above in order to fit most network types ?