Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The common approach for training neural networks, as far as i know, is the backpropagation algortihm, which uses gradient descent to reduce the error.

(i) why should one use a fixed learning rate / simulated annealing over, let's say, Armijo rule?

(ii) is there a good reason to use gradient descent in this case, or did that just grow historically? Especially, is gradient descent in this case favorable over algorithms like Newton's (Quasi-Newton, globalized Newton)?

share|cite|improve this question
up vote 2 down vote accepted

Backpropagation algorithm IS gradient descent and the reason it is usually restricted to first derivative (instead of Newton which requires hessian) is because the application of chain rule on first derivative is what gives us the "back propagation" in the backpropagation algorithm. Now, Newton is problematic (complex and hard to compute), but it does not stop us from using Quasi-newton methods especially BFGS (I believe many neural network software packages already use BFGS as part of their training these days).

As for fixed learning rate, it need not be fixed at all. There are papers far back as '95 reporting on this (Search for "adaptive learning rate backpropagation").

share|cite|improve this answer
Newbob training is an example of adaptive learning rates. I explain it in my bachelor's thesis (will be online in a month or so). – Martin Thoma Nov 6 '14 at 11:20
You should point out that backpropagation is a (real) subset of all gradient descent algorithms. It is the subset where a training rule exists that is recursively defined (hence the name _back_propagation). – Martin Thoma Nov 6 '14 at 11:45
@moose I will have to think about your suggestion carefully. That would require me to elaborate on the answer a little more than I would like to do, but let me see. – TenaliRaman Jun 3 '15 at 16:32

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.