# Using gradient descent: cost decreases, then increases

I am minimizing a function using gradient descent. The learning rate is fixed. First, for few iterations, the cost decreases; after that, it starts increasing. What is the reason for this?

• Problem-specific. Decreases and increases how? If the energy goes like 10, 1, 0, 0.01, 0.02, 0.03, 0.04, ... it's likely to be numerical error. If it goes like 10, 1, 0, 5, 10, 100, then you're not doing gradient descent on a good function.
– snar
Oct 15, 2014 at 15:59
• Try reducing the learning rate slowly, e.g. $\sim 1/\sqrt{k}$ or $\sim 1/\log k$. Oct 15, 2014 at 16:01
• If learning rate = step size, then you should use a linesearch method to determine step size.
– daw
Oct 15, 2014 at 17:58

Hessian encodes the second derivatives of a function with respect to all pairs of variables. So, if there are n inputs to a function, the gradient is n-dimensional and the Hessian is nxn-dimensional. In machine learning, the inputs are usually the features and the function is usually a loss function we are trying to minimize. When the Hessian is ill-conditioned, it means the basis of the loss function have contours that are very long “ellipsoids” rather than being close to “circular”. This causes problems for first-order optimization methods like gradient descent (w=w−η∇w) which need to follow a very zigzag path to the minimum. The first figure below shows the gradient directions at various points on a straight line through the parameter space. Notice how the directions away from the center are nearly orthogonal to the useful direction of descent. The second figure shows the zigzag path gradient descent has to take if the Hessian is ill-conditioned, the function contours are stretched out, and the gradients often point to directions that might not be the best way to descend to the minimum.