I am looking for methods to improve the gradient descent algorithm. In particular, my problem is that i'm trying to minimise the sum of squared errors between some observed data and the data produced by my model (based on non-linear differential equations). I'm using gradient descent to find a better fit. But the problem is that the errors are very large due to the nature of the inputs involved. This is leading to both a very slow convergence and an overflow of the variables. So my questions are:

Is there any way to ensure that i do not have to deal with such large numbers. Like scaling or taking logarithms?

Are there other ways to get faster convergence from the algorithm?

Thanks for the help.

  • $\begingroup$ Your question is unclear because of "overflow of the variables". Do you have some $\exp$ in your objective function, why/where ? You are supposed to start with a toy problem and make it work before going to the real problem. Stochastic gradient descent contains everything we can say in general, but for particular problems, there are much more useful modifications. $\endgroup$ – reuns Dec 5 '17 at 14:46
  • $\begingroup$ I don't have any exponentials in my code. It's just i have numbers as large as 10^6 and so it's resulting in large error values. I guess i'll try with dummy values first to check like you said. $\endgroup$ – Hikaru Dec 5 '17 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.