0
$\begingroup$

In many gradient descent algorithms to backpropagate an error through a neural network the final line looks something like this:

$$ W_{ij} = W_{ij} - \mu \frac{\delta E}{\delta W} $$

i.e. adjust the weights by an amount $\mu$ with magnitude proportional to the gradient of the error with respect to the weights. I have looked quite hard for a good explanation of why this works but I cannot find it anywhere- even some of the papers that defined backpropagation in the 1980s, e.g. by Werbos, Pineda, Hecht-Nielson, only quote this result without explaining it.

In particular, in Pineda 1987, which I believe is the first place the backpropagation algorithm was generally defined, it is stated that the gradient of the weights is directly proportional to he gradient of the error:

$$ \frac{\delta W_{ij}}{\delta t} = -\mu \frac{\delta E}{\delta W_{ij}} $$

And Pineda justifies this by stating, to ensure that the output of the network converges towards the target values, "let the system evolve in the weight space long trajectories that are antiparallel to the gradient of E."

Perhaps there is a simple explanation that I a overlooking as to why this would ensure the convergence?

$\endgroup$
1

2 Answers 2

0
$\begingroup$

It works because we combine three things:

  1. An objective we want to minimize ( error squared ).
  2. A differentiable function (often a sigmoid) and
  3. The chain rule of differentiation.

A Neural network is a series of linear combinations of 2). By linearity of the differentiation operation we can apply the chain rule on such linear combinations.

$\endgroup$
0
$\begingroup$

The paper referenced in SagarM's reply is a good place to look for a derivation, but it doesn't quite answer the question I had in mind however- it did very much help me realise what I didn't understand though, so look there first if you are having a similar question to the one I asked above).

Specifically, in many derivations of the backpropagation algorithm (especially, it seems, by people who are not fluent with the literature in this field) the starting point for explaining why the above algorithm works is a Taylor series expansion. And from this, there is often a jump in logic to say that we can then write the above algorithm. Something like this:

$$ F(x_{0} + \delta) = F(x_{0}) + \delta \frac{\delta F}{\delta x}_{x_{0}} + \delta^{2} \frac{\delta^{2}F}{\delta x^{2}}_{x_{0}} + ... $$

then in the limit of small $\delta$s we only have the linear term. From this, analogously it is then written:

$$ W_{ij} = W_{ij}^{0} - \delta \frac{\delta E}{\delta W}_{W_{}ij^{0}} $$

Where the $\delta$ is made negative because we are moving in the weight space to decrease E.

Now, the way I interpret the gradient term $\frac{\delta E}{\delta W_{ij}}$ in the equation above is as a term that simply indicate which direction to move the weights in, i.e. it does not actually tell you anything directly about $\frac{\delta W_{ij}}{\delta x_{k}}$, where k can be any variable that $W_{ij}$ depends on.

I haven't seen any derivation that makes this subtle but crucial point clear, and I have spent quite a lot of time trying to derive the equivalence of $\frac{\delta E}{\delta W_{ij}}$ to $\frac{\delta W_{ij}}{\delta x_{k}}$. Hopefully my answering my own question can help others understand this more easily in future.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .