# Backpropagation in recurrent neural networks

How do recurrent neural networks share weights ? I have been reading it online but I cant figure out how it does this. Particularly because during backpropagation,the hidden cell at e.g. t=2 would receive the gradients coming from t=3. So both now the weights of both cells will be updated differently.

For example, after 1 iteration, will the weights w1,w2,w3,w4,w5 be similar ? Because w4 would have gradients coming from its output as well as the output from w5 whereas w5 would only have the gradient coming from its output.

• Run it with your input, compute as usual (using backpropagation) the error and the gradient with respect to the weights $\frac{\partial E}{\partial W_{i,j}(t)}$,
• Finally update the weights $W_{i,j}$ of your recurrent neural network with $$W_{i,j} \leftarrow W_{i,j} - \eta \sum_t \frac{\partial E}{\partial W_{i,j}(t)}$$
• @kong You need to understand the unroll step which sets $W_{i,j}(t) = W_{i,j}$ for every $t$. At the end we update $W_{i,j}$ (equivalently each $W_{i,j}(t)$ are updated the same way) so $W_{i,j}(t) = W_{i,j}$ always stays true. What depends on $t$ is $\frac{\partial E}{\partial W_{i,j}(t)}$ – reuns Oct 20 '17 at 19:56
• @kong Yes exactly. And I didn't mention a main issue : what $t_{max}$ do you choose, and how do you define the error in the unrolled network (do you look only at the output of the last layer, or do you sum the error of the output of the last few layers) – reuns Oct 20 '17 at 20:29