# Using gradient descent: cost decreases, then increases

I am minimizing a function using gradient descent. The learning rate is fixed. First, for few iterations, the cost decreases; after that, it starts increasing. What is the reason for this?

• Problem-specific. Decreases and increases how? If the energy goes like 10, 1, 0, 0.01, 0.02, 0.03, 0.04, ... it's likely to be numerical error. If it goes like 10, 1, 0, 5, 10, 100, then you're not doing gradient descent on a good function.
– snar
Oct 15, 2014 at 15:59
• Try reducing the learning rate slowly, e.g. $\sim 1/\sqrt{k}$ or $\sim 1/\log k$. Oct 15, 2014 at 16:01
• If learning rate = step size, then you should use a linesearch method to determine step size.
– daw
Oct 15, 2014 at 17:58