I am reading a paper explaining the derivations of the back-propagation equations in RNNs. There I read 'Note that the Weight Matrix remains the same across all time sequence so we can differentiate to it at each time step and sum all together.'

enter image description here

My question is why this statement is correct. What is its mathematical derivation?

Your advice will be appreciated.

  • $\begingroup$ If your objective function is $|f(t_{end})-y|^2$ where $x$ is the input, $f_t(x)$ the output at time $t$, and $y$ the desired output then it means you don't care of the output at $t < t_{end}$, but you do care of the partial derivative $ \frac{\partial f(t_{end})}{\partial z_{j,t}}$ where $z_{j,t}$ is output of $j$-th neuron at time $t$. Also your RNN will be quite similar to a $t_{end}$-layer non-recurrent NN, and it is not really possible to train a NN with too many layers. The alternative is to choose objective function $\sum_t|f_t(x)-y_t|^2$ possibly with constant $y_t = y_0$. $\endgroup$ – reuns Apr 20 at 8:46
  • $\begingroup$ The weights don't depend on $t$ so you'll have $\frac{\partial E}{\partial w_{i,j}} = \sum_t \sum_{t'\le t} \frac{\partial |f_t(x)-y_t|^2 }{\partial z_{j,t'}(x)}$ $\endgroup$ – reuns Apr 20 at 8:49
  • $\begingroup$ Thank you this helps. Two questions: 1) In your second comment you meant to write w rather than z ? (the derivative is w.r.t. W). 2) What if we have a RNN modeling a many to one sequence (e.g., sentiment analysis task) where the error is calculated in the end only? $\endgroup$ – AlK Apr 20 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.