Specifically, I am trying to go backwards with a neural network to have the program draw what it views as the ideal image of a certain shape.
Forward propagation in my neural network is done through:
$L_n$=$\sigma(W_n\cdot L_{n-1}+B_n$)
Where L, W and B are all non-square matrices.
Going backwards, I came up with the equation:
$L_{n-1} = W_n^{-1}\cdot (\sigma^{-1}(L_n)-B_n)$
However, as I am now learning, $W_n$ cannot have an inverse as its dimensionality is determined by $L_n$'s rows and $L_{n-1}$'s columns and as a result it cannot be square since two layers will not have the same number of columns as the other has rows. Given the first equation, is there anyway for me to find (or to at least approximate) $L_{n-1}$ from $L_n$? I am a grade 12 student and have no formal knowledge of linear algebra. Everything I know about it or matrices I know from my venture into neural networks. As such I'd appreciate simple answers as I will probably not be able to understand anything beyond a high school level or what I have shown in this post.
There are probably multiple matrices $L_{n-1}$ that would lead to the same $L_n$ and that is fine.I just want to be able to find any one of those possible matrices. As for further clarification of what is happening: The layer matrices are all n rows by 1 column where the individual elements are neurons. The number of neurons in different layers varies. Layer one in my case contains 1024 neurons, while layer 2 and 3 contain 8, and while layer 4 contains 2 neurons. The weights determine the strength of connections from layer to layer. For example, $W_2$ determines the strengths of connections from $L_1$ to $L_2$
In the aforementioned equation: $L_n$=$\sigma(W_n\cdot L_{n-1}+B_n$)
$L_2$ is defined based on the connections from $L_1$ to $L_2$ and the actual input that is fed into $L_1$
This feed forward model is used to get the input in the first layer and spit out an output in the final layer. For example, if I show the program a picture of an "X" (I'm doing Xs and Os) the first layer will light up and the final layer will (if the network has been trained) display some high number in the neuron representing an answer of X and some low number in the neuron representing an answer of O. My objective is to set $L_4$ (the last layer) to something like
\begin{matrix} 0.99 \\ 0.01 \\ \end{matrix} Then go backwards to see what input would have caused this output. This should allow me to visualize what the computer's ideal X or O shape would be, and allow it to draw said shape with $L_1$

  • $\begingroup$ How is defined, in terms of components, the product $W\cdot L$? $\endgroup$ – Emilio Novati Mar 2 '18 at 17:44
  • 1
    $\begingroup$ Welcome to stackexchange, and welcome to mathematical research. That said, there may not be a useful answer to your question without going deeper into linear algebra than you'd like. In particular, you may not be able to "go backwards" in a unique way if there are different values of $L_{n-1}$ that can lead to the same value of $L_n$. If that's the case you may have to specify what kind of "approximation" you might want. You are more likely to get some help here if you edit the question to include a small numerical example of what you have in mind. $\endgroup$ – Ethan Bolker Mar 2 '18 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.