I want to use Armijo Rule to find suitable learning rate for my neural network.
The network is a multiclass classification. It has one hidden layer. Input to hidden layer uses Sigmoid filter, and hidden to output uses softmax.
How do I apply Armijo algorithm here? Do I calculate Armijo for each backpropagation step (out to hidden, and then hidden to input), or do I calculate Armijo at once for entire network? The latter seems more complicated, since I need a derivative (gradient) for calculating Armijo step. I do have gradients for each backstep, but how do I find the gradient for entire network?
Also, calculating Armijo for entire network seems intuitively more costly, since I have to calculate entire network each time.