I am following Pattern Recognition and Machine Learning by Christopher Bishop chapter 5 on Neural Networks and I do not fully understand the formulas for backpropagation in order to calculate the Jacobian.

Consider a Network with 2 input units $x_{i}$, 2 hidden units, and 2 output units $y_{k}$.

We want to solve:

$J_{ki} = \frac{\partial y_{k}}{\partial x_{i}} = \sum_{j} \frac{\partial y_{k}}{\partial a_{j}} \frac{\partial a_{j}}{\partial x_{i}} $

The sum over $j$ is a sum over the 2 hidden units. Hence $\frac{\partial y_{k}}{\partial a_{j}}$ measures the change in output $y_{k}$ due to a change in the activation $a_{j}$ of hidden unit $j$ and $ \frac{\partial a_{j}}{\partial x_{i}}$ measures the change in the activation $a_{j}$ due to a change in the input. As $a_{j} = \sum_{i} w_{ji}x_{i}$, $ \frac{\partial a_{j}}{\partial x_{i}}$ is clearly $w_{ji}$.

Hence we have:

$J_{ki} = \sum_{j} \frac{\partial y_{k}}{\partial a_{j}} \frac{\partial a_{j}}{\partial x_{i}} = \sum_{j} \frac{\partial y_{k}}{\partial a_{j}} w_{ji}$

Now is where I cannot completely follow the formulas. Above we see that we want to calculate $\frac{\partial y_{k}}{\partial a_{j}}$ for each hidden unit. It is given by the following formula:

$\frac{\partial y_{k}}{\partial a_{j}} = \sum_{l} \frac{\partial y_{k}}{\partial a_{l}} \frac{\partial a_{l}}{\partial a_{j}}$

I do not understand why we are summing over $l$ and what $a_{l}$ should be. If $a_{j}$ is a single hidden unit and $y_{k}$ is a single output unit, why would we need a sum to measure the relationship between these two. Bishop says that "the sum runs overr all units $l$ to which $j$ sends connections".



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.