I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).

Suppose that the network is simple like so (forward pass):

$$\begin{aligned}z_1 &= xW_1 + b_1 & (1) \\ a_1 &= \tanh{(z_1)} & (2) \\ z_2 & = a_1W_2 + b_2 & (3)\\ a_2 & = \text{softmax}(z_2)=\hat{y}_n & (4) \\ l_n & = \log{(\hat{y}_n)} & (5) \\ PL_n & = l_n\dot{y}_n^T & (6) \\ L & = -\frac{1}{N} \sum_{n \in N} PL_n & (7) \end{aligned} $$

So only one hidden layer. Let the dimensions be the following:

$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.

Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):

$$\begin{aligned} dPL_n & = -1/N & (7) \\ d\dot{y}^T_n & = l_n^T dPL_n & (6) \\ dl_n & = \dot{y}^T_n dPL_n & (6) \\ d\hat{y}_n & = 1/\hat{y}_n & (5) \end{aligned} $$

Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?

(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).

Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(

I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.

I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2 = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &= \frac{\partial l_n}{\partial z_2} dl_n \\ &= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$

I think $dz_2$ cannot be of the same dimesion as $z_2$... it has to be $2 \times 2$

Same question asked on DataScience.SE.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.