This is a basic recurrent neural network (RNN), and I am trying to figure out the derivation of its backpropagation and all intermediate steps, I encounter 3rd-order tensor, and not sure if I am doing right.
This RNN is designed for a language model, so its purpose is to predict the next word given previous words in a sentence. The input word is represented as a word vector. The cost function is cross-entropy ($CE$).
The definition:
\begin{align*} \underset{1 \times D_h}{h^{(t)}} &= \mathrm{sigmoid}(\underset{1 \times D_h}{z_1^{(t)}}) = \mathrm{sigmoid}\bigg( \underset{1 \times D_h,}{h^{(t - 1)}} \underset{D_h \times D_h}{H} + \underset{1\times d,}{e^{(t)}} \underset{d \times D_h}{I} + \underset{1 \times D_h}{b_1} \bigg) \\ \underset{1 \times \left|V\right|}{\hat y^{(t)}} &= \mathrm{softmax}(\underset{1 \times \left|V\right|}{z_2^{(t)}}) = \mathrm{softmax}\bigg( \underset{1 \times D_h,}{h^{(t)}} \underset{D_h \times \left| V \right |}{U} + \underset{1 \times \left| V \right|}{b_2} \bigg) \\ J^{(t)} &= CE(y^{(t)}, \hat{y}^{(t)}) = \sum_{j=1}^{\left|V\right|}y_j^{(t)}\mathrm{log}(\hat{y}_j^{(t)}) \end{align*}
- $e^{(t)}$: the input word embedding at $t$th time step.
- $I$: the input word representation matrix.
- $H$: the hidden transformation matrix.
- $U$: the output word representation matrix.
Dimensions:
- $\left|V\right|$: dimension of vocabulary
- $D_h$: hidden layer size
- $d$: size of word embedding
$z_1$ and $z_2$ are intermeidate variables added for convenience.
As seen, I labeled the matrix sizes for each quantity except for scalar.
Here, I am most interested in the derivation of
$$\underset{D_h \times D_h}{\frac{\partial J^{(t)}}{\partial H}\bigg|_t} $$
with emphasis on all the intermediate steps. Here is my attempt:
\begin{align*} \underset{D_h \times D_h}{\frac{\partial J^{(t)}}{\partial H}\bigg|_t} &= \underset{1 \times \left|V\right|,}{\frac{\partial J^{(t)}}{\partial z_2^{(t)}}} \underset{\left|V\right| \times D_h,}{\frac{\partial z_2^{(t)}}{\partial h^{(t)}}} \underset{D_h \times D_h,}{\frac{\partial h^{(t)}}{\partial z_1^{(t)}}} \underset{D_h \times D_h \times D_h}{\frac{\partial z_1^{(t)}}{\partial H}} \\ &= \underset{1 \times \left|V\right|,}{(\hat{y}^{(t)} - y^{(t)})} \underset{\left|V\right| \times D_h,}{U^T} \underset{D_h \times D_h,}{diag\{h^{(t)} \circ (1 - h^{(t)})\}} ? \\ &= \underset{D_h \times 1}{(h^{(t - 1)})^T} \underset{1 \times \left|V\right|,}{(\hat{y}^{(t)} - y^{(t)})} \underset{\left|V\right| \times D_h,}{U^T} \underset{D_h \times D_h,}{diag\{h^{(t)} \circ (1 - h^{(t)})\}} \end{align*}
The parts that I've already figured out:
- $(\hat{y}^{(t)} - y^{(t)})$ is the derivative of $J$ w.s.t. the input to softmax ($z_2$)
- $diag\{\cdots\}$ takes advantage of the sigmoid function whose derivative has the form of $\sigma(1 - \sigma)$.
But I am still not sure of the second equality because I don't know how to deal with $\frac{\partial z_1^{(t)}}{\partial H}$ (where I put a question mark), which is the derivative of a vector w.s.t. a matrix, and the result seems to be a 3rd-order tensor ($D_h \times D_h \times D_h$).
I checked the solution and the last equality is likely to be correct. But the transpose and then left multiply of $(h^{(t - 1)})^T$ looks too magical to me. Can anybody shed some light on this part, please?
I also put the graph here in case anyone is interested in looking at it. All quantities are labeled consistently with the above equations.
