Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I'm working on understanding all the math used in artificial neural networks. I have gotten stuck at calculating the error function derivatives for hidden layers when performing backpropagation.

On page 244 of Bishop's "Pattern recognition and machine learning", formula 5.55. The derivative of the error function for a hidden layer is given using a sum of derivatives over all units to which it sends connections.

$$ \frac{\partial E_n}{\partial a_j} = \sum_k \frac{\partial E_n}{\partial a_k} \frac{\partial a_k}{\partial a_j}$$

I know the chain rule. If $a_j$ goes into only one other node, we can apply the chain rule to separate the parts. But what is the intuition behind summing these values for all nodes if the output goes into multiple nodes?

Thanks

share|improve this question
    
It makes no sense with total derivatives. In the book it's written with partial derivatives. The $\TeX$ command for the partial derivative symbol is \partial. There's an edit link underneath the question. It might also be a good idea to link to the book in the question so people can look up the context. –  joriki Feb 20 '13 at 18:22
    
Thanks, I changed the derivatives to partial. –  Marek Feb 20 '13 at 22:36
    
The formula can also be seen here: web.cs.swarthmore.edu/~meeden/cs81/s10/BackPropDeriv.pdf However, I haven't found any rule or explanation why this summing works and is correct. –  Marek Feb 20 '13 at 22:43
    
This is backpropagation, so the error information is flowing backward through the network. The index $k$ iterates over all the nodes to which node $j$ supplies information in the forward sense; intuitively, error in node $a_k$ for each index $k$ is due, in some part, to the action of node $j$. –  Arkamis Feb 20 '13 at 22:44
    
Yes, I understand that. But why does this sum work? How does summing over the right side equal the left side? Is there a rule for that? I tried to derive this on my own and hit a dead end when the signal is sent into multiple nodes. –  Marek Feb 20 '13 at 22:52
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.