I want to use Armijo Rule to find suitable learning rate for my neural network.

The network is a multiclass classification. It has one hidden layer. Input to hidden layer uses Sigmoid filter, and hidden to output uses softmax.

How do I apply Armijo algorithm here? Do I calculate Armijo for each backpropagation step (out to hidden, and then hidden to input), or do I calculate Armijo at once for entire network? The latter seems more complicated, since I need a derivative (gradient) for calculating Armijo step. I do have gradients for each backstep, but how do I find the gradient for entire network?

Also, calculating Armijo for entire network seems intuitively more costly, since I have to calculate entire network each time.


This paper: http://www.dcs.bbk.ac.uk/~gmagoulas/AdaptAlgorithms.pdf describes the process quite neatly.

From what I understand, you must flatten weight and biases and concatenate into one huge parameter vector which is treated as the input to your overall loss function. Since you have already calculated the derivatives for the loss function to every parameter, you can apply Armijo Rule to get a common learning rate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.