# Finding partial derivative of a matrix

In the image replace s with h

$$A$$: $$n\times{h}$$ matrix of hidden layer inputs

$$X$$: $$n\times{d}$$ matrix of data inputs of dimensionality d

$$U$$: $$d\times{h}$$ matrix

$$H$$: $$n\times{h}$$ matrix of hidden layer outputs for each batch, at each timestep

$$W$$: $$h\times{h}$$ matrix

$$V$$: $$h\times{v}$$ matrix

$$b$$: $$1\times{h}$$ bias term

In the recurrent neural network, input to the hidden state output at timestep is defined as:

$${A}_{n\times h}=X_{n\times{d}}U_{d\times{h}} + H_{n\times{h}}W_{h\times{h}} + \bf b$$

Here is the same equation but with timestep notation for clarity:

$${A}^{(t)}=X^{(t)}\,U + H^{(t-1)}W + \bf b$$

Hidden layer output of each time step is a tangens hyperbolic activation of its inputs:

$$H^{(t)}=tanh(A^{(t)})$$

which is used to calculate the output $$o^{(t)}$$ for each of the hidden states $$h_i^{(t)}$$, and to pass on to the next hidden state in the time series (could be imagined as passing to the right state in the figure).

$$o^{(t)} = H^{(t)}V + c$$

$$\hat{y}_j^{(t)} = softmax(o^{(t)}) = \frac{o_j^{(t)}}{\sum_{k} o_k^{(t)}}$$

L is a cross entropy loss function, and $$y$$ is the correct output vector

$$L = - \frac{1}{T} \sum_{t=1}^{T}\sum_{j=1}^{v} \hat{y}_{tj} \times log(y_{tj})$$

During the backpropagation we need to calculate derivative of the loss w.r.t parameters U, W, V, b, c.

Given $$\frac{\partial{L}}{\partial{A}}$$ of size $$n\times{h}$$ The derivative of L w.r.t parameter U is :

$$\frac{\partial{L}}{\partial{U}} = X^{T} \frac{\partial{L}}{\partial{A}}$$

How do i derive this result? What are the rules and operators i need to know to produce this.

• @RodrigodeAzevedo I updated my question May 14 '19 at 16:59

To reduce unnecessary clutter, drop the $$t$$-superscripts, use lowercase letters for vectors and uppercase for matrices. Then write the differential of $$L$$ in terms of $$da$$ and perform a change of variables to $$dU$$ \eqalign{ dL &= \frac{\partial L}{\partial a}:da \cr &= \frac{\partial L}{\partial a}:x\,dU \cr &= x^T\Big(\frac{\partial L}{\partial a}\Big):dU \cr \frac{\partial L}{\partial U} &= x^T\Big(\frac{\partial L}{\partial a}\Big) \cr } since $$(x,a)$$ are row vectors, the resulting gradient is a $$d\times h$$ matrix.
NB: A colon denotes the trace/Frobenius product, i.e. $$A:B = {\rm Tr}(A^TB)$$