In a machine learning course, the professor described the gradient descent method for calculating the regression line. Shortly, we're looking for the $a$ and $b$ in $y=ax+b$ which describes the line. A cost function $E$ measures the distance between the line and all the points. Thus, we have a surface $E(a,b)$, and finding the minimum of $E$ gives the best line function.

There are, however, statistical methods for calculating regression lines, for example as shown here, using the standard deviation of $X$, $Y$ and Pearson's coefficients.

Are the two methods equivalent? Do they give the same results? When should we use each?

  • $\begingroup$ What is your noise model? $\endgroup$ – Rodrigo de Azevedo Apr 22 at 8:40
  • 2
    $\begingroup$ The gradient descent method should be used when you cannot explicitly calculate the $a$ and $b$ which minimize your problem. If you have a simple enough problem, as in the video you have linked, it's unnecessary to use optimization algorithms like gradient descent to "search" for the best $a$ and $b$ since you can explicitly write them down as functions of the data (e.g. standard deviation, pearson's coefficients, etc). $\endgroup$ – Tony S.F. Apr 22 at 8:43
  • 1
    $\begingroup$ The two methods give a solution to the same optimization problem (least squares). What you call "statistical methods" are just explicit formulas for the optimum, while the gradient descent is a numerical method that converges to the solution (there are many other numerical methods btw). So yes, they are equivalent. Note that an explicit formula is available for linear least squares, but for other regression methods it's often necessary to rely on numerical methods. $\endgroup$ – Jean-Claude Arbaut Apr 22 at 8:51
  • $\begingroup$ @RodrigodeAzevedo I'm mathematically naive. I don't know about a noise model :( $\endgroup$ – Avi Apr 22 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.