Say I want to fit a straight line using Total Least Square (as opposed to Least Square), which is to minimize the sum of (yi-k*xi-b)^2/(k^2+1) over all xi's and yi's, where xi's and yi's are training data point coordinates, and k and b are the fitting parameters.

I use gradient descent to solve for k and b. The cost function is just the above sum. But the result turns out that the cost isn't monotonic, it goes down and goes up. It's not supposed to be so, as this sum has 1 global minimum only. I don't know what goes wrong to make the cost function non monotonic?

  • $\begingroup$ How could we see what goes wrong in your calculus without without having your calculus in detail ? $\endgroup$
    – JJacquelin
    Dec 10, 2018 at 14:25
  • $\begingroup$ Even for a simple quadratic function, gradient descent isn't guaranteed to converge if you don't pick a stepsize small enough. Consider for instance $f(x)=x^2$. If your stepsize is $h=1$, then $x_{n+1}=x_n-hf'(x_n)=x_n-2x_n=-x_n$ simply oscillates and never reaches the unique minimum. $\endgroup$
    – Federico
    Dec 10, 2018 at 14:31
  • $\begingroup$ @ JJacquelin I guarantee the rest of my calculus and gradient descent is correct. May you confirm that the above sum is correct? $\endgroup$
    – feynman
    Dec 13, 2018 at 7:00
  • $\begingroup$ @ Federico I guarantee my gradient descent calculation is correct, and step lengths are nothing to worry about because I use matlab ode23 package to solve. I didn't set the step lengths. $\endgroup$
    – feynman
    Dec 13, 2018 at 7:01

1 Answer 1


In applied statistics, total least squares is a type of errors-in-variables regression, a least squares data modeling technique in which observational errors on both dependent and independent variables are taken into account. It is a generalization of Deming regression and also of orthogonal regression, and can be applied to both linear and non-linear models.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .