# How to calculate the Lie algebra of a neural network?

Define $F$ as the standard multi-layer feed-forward perceptron: $$F(\mathbf{x}) = \Theta( W_1 \circ \Theta( W_2 \circ .... W_L(\mathbf{x})))$$ where $\Theta$ is the sigmoid function and $W_\ell$ is the weight matrix for layer $\ell$, every layer contains $N$ neurons, with a total of $L$ layers.

$x \in X$ is a vector space of dimension $N$.

$F$ forms a group acting on $X$.

$F$ can be parameterized by the $N \times N$ weight matrices $W_\ell$ (there are $L$ of them). The total number of parameters is $L \cdot N^2$. In other words, $F$ is parameterized by $W \in \mathbb{R}^{L N^2}$.

How can I calculate the explicit elements of the Lie algebra $\mathfrak{g} = \mbox{Lie } F$?

The sigmoid function can be either: $$\Theta(\xi) = \frac{1}{1+ e^{-a \xi}} \quad \mbox{(logistic)}$$ $$\Theta(\xi) = \tanh(\xi)$$ $$\Theta(\xi) = \begin{cases} 0, & \xi < 0 \\ \xi, & \xi \geq 0 \end{cases} \quad \mbox{(rectifier)}$$ (choose whichever one leads to an elegant solution).

I'm just beginning to learn Lie theory, and would like to apply it to neural network research. Thanks.

• This could use some clarification - you seem to be conflating a single transformation $F$ (as you defined it) with the group of all such transformations. What are the allowed values of the $W_\ell$? You certainly need at least invertibility if you want these transformations to form a group. How does the function $\Theta : \mathbb R \to \mathbb R$ act on $X$? Componentwise after fixing a basis? Also $F \in \mathbb R^{L N^2}$ does not mean what you intend. – Anthony Carapetis Nov 7 '15 at 9:19
• Technically you don't need $\Theta$ to be self-inverse, but it certainly needs to be invertible, and no composition of matrices and $\tanh$ is going to give you $\tanh^{-1}$. The first thing to check is the presence of an identity transformation, which I think might already force $\Theta$ to be linear. Certainly does in the case $L=1$. – Anthony Carapetis Nov 7 '15 at 9:53
• @YanKingYin Can you tell me which textbooks you're using to learn this material? Any recommendations? – littleO Nov 7 '15 at 10:27
• If $\Theta(\xi) = \xi$ then $L$ doesn't matter, since the product of two matrices is a matrix. You'd just get $GL(N)$ (assuming you require invertibility of $W$). – Anthony Carapetis Nov 7 '15 at 14:40
• littleO: "Matrix groups for undergraduates" may be a nice and gentle introduction. – Yan King Yin Nov 11 '15 at 15:10