enter image description here

In the image replace s with h

$A$: $n\times{h}$ matrix of hidden layer inputs

$X$: $n\times{d}$ matrix of data inputs of dimensionality d

$U$: $d\times{h}$ matrix

$H$: $n\times{h}$ matrix of hidden layer outputs for each batch, at each timestep

$W$: $h\times{h}$ matrix

$V$: $h\times{v}$ matrix

$b$: $1\times{h}$ bias term

In the recurrent neural network, input to the hidden state output at timestep is defined as:

$$ {A}_{n\times h}=X_{n\times{d}}U_{d\times{h}} + H_{n\times{h}}W_{h\times{h}} + \bf b $$

Here is the same equation but with timestep notation for clarity:

$$ {A}^{(t)}=X^{(t)}\,U + H^{(t-1)}W + \bf b $$

Hidden layer output of each time step is a tangens hyperbolic activation of its inputs:


which is used to calculate the output $ o^{(t)} $ for each of the hidden states $ h_i^{(t)} $, and to pass on to the next hidden state in the time series (could be imagined as passing to the right state in the figure).

$$ o^{(t)} = H^{(t)}V + c $$

$$ \hat{y}_j^{(t)} = softmax(o^{(t)}) = \frac{o_j^{(t)}}{\sum_{k} o_k^{(t)}} $$

L is a cross entropy loss function, and $y$ is the correct output vector

$$ L = - \frac{1}{T} \sum_{t=1}^{T}\sum_{j=1}^{v} \hat{y}_{tj} \times log(y_{tj}) $$

During the backpropagation we need to calculate derivative of the loss w.r.t parameters U, W, V, b, c.

Given $ \frac{\partial{L}}{\partial{A}} $ of size $ n\times{h} $ The derivative of L w.r.t parameter U is :

$$ \frac{\partial{L}}{\partial{U}} = X^{T} \frac{\partial{L}}{\partial{A}} $$

How do i derive this result? What are the rules and operators i need to know to produce this.

  • $\begingroup$ @RodrigodeAzevedo I updated my question $\endgroup$
    – monolith
    May 14 '19 at 16:59

To reduce unnecessary clutter, drop the $t$-superscripts, use lowercase letters for vectors and uppercase for matrices. Then write the differential of $L$ in terms of $da$ and perform a change of variables to $dU$ $$\eqalign{ dL &= \frac{\partial L}{\partial a}:da \cr &= \frac{\partial L}{\partial a}:x\,dU \cr &= x^T\Big(\frac{\partial L}{\partial a}\Big):dU \cr \frac{\partial L}{\partial U} &= x^T\Big(\frac{\partial L}{\partial a}\Big) \cr }$$ since $(x,a)$ are row vectors, the resulting gradient is a $d\times h$ matrix.

NB: A colon denotes the trace/Frobenius product, i.e. $$A:B = {\rm Tr}(A^TB)$$

  • $\begingroup$ I'm sorry i'm not familiar with these terms, write differential in terms of, perform change of variables, also this trace operator, why use it? Thank you... $\endgroup$
    – monolith
    May 14 '19 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.