9
$\begingroup$

Define $F$ as the standard multi-layer feed-forward perceptron: \begin{equation} F(\mathbf{x}) = \Theta( W_1 \circ \Theta( W_2 \circ .... W_L(\mathbf{x}))) \end{equation} where $\Theta$ is the sigmoid function and $W_\ell$ is the weight matrix for layer $\ell$, every layer contains $N$ neurons, with a total of $L$ layers.

$x \in X$ is a vector space of dimension $N$.

$F$ forms a group acting on $X$.

$F$ can be parameterized by the $N \times N$ weight matrices $W_\ell$ (there are $L$ of them). The total number of parameters is $L \cdot N^2$. In other words, $F$ is parameterized by $W \in \mathbb{R}^{L N^2}$.

How can I calculate the explicit elements of the Lie algebra $\mathfrak{g} = \mbox{Lie } F$?

The sigmoid function can be either: $$\Theta(\xi) = \frac{1}{1+ e^{-a \xi}} \quad \mbox{(logistic)}$$ $$\Theta(\xi) = \tanh(\xi)$$ $$\Theta(\xi) = \begin{cases} 0, & \xi < 0 \\ \xi, & \xi \geq 0 \end{cases} \quad \mbox{(rectifier)}$$ (choose whichever one leads to an elegant solution).

I'm just beginning to learn Lie theory, and would like to apply it to neural network research. Thanks.

$\endgroup$
  • 2
    $\begingroup$ This could use some clarification - you seem to be conflating a single transformation $F$ (as you defined it) with the group of all such transformations. What are the allowed values of the $W_\ell$? You certainly need at least invertibility if you want these transformations to form a group. How does the function $\Theta : \mathbb R \to \mathbb R$ act on $X$? Componentwise after fixing a basis? Also $F \in \mathbb R^{L N^2}$ does not mean what you intend. $\endgroup$ – Anthony Carapetis Nov 7 '15 at 9:19
  • 1
    $\begingroup$ Technically you don't need $\Theta$ to be self-inverse, but it certainly needs to be invertible, and no composition of matrices and $\tanh$ is going to give you $\tanh^{-1}$. The first thing to check is the presence of an identity transformation, which I think might already force $\Theta$ to be linear. Certainly does in the case $L=1$. $\endgroup$ – Anthony Carapetis Nov 7 '15 at 9:53
  • 1
    $\begingroup$ @YanKingYin Can you tell me which textbooks you're using to learn this material? Any recommendations? $\endgroup$ – littleO Nov 7 '15 at 10:27
  • 1
    $\begingroup$ If $\Theta(\xi) = \xi$ then $L$ doesn't matter, since the product of two matrices is a matrix. You'd just get $GL(N)$ (assuming you require invertibility of $W$). $\endgroup$ – Anthony Carapetis Nov 7 '15 at 14:40
  • 1
    $\begingroup$ littleO: "Matrix groups for undergraduates" may be a nice and gentle introduction. $\endgroup$ – Yan King Yin Nov 11 '15 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.