In this Khan Academy video series Khan goes through the derivation of the formula for the linear regression line for some data points.

The only part I do not understand is the one I've given a link to. Particularly, I don't understand why Khan is so sure that when he sets the partial derivatives to zero, he is going to get the squared error function at its minimum (as opposed to its maximum).

How does he know that? He doesn't explain this in the video, so I believe it must be more or less obvious.

A short answer explaining this in simple terms would be much appreciated.

  • $\begingroup$ The short answer convexity. The longer answer: one can prove it with Cauchy Schwarz inequality. Google ordinary least squares and normal equation. $\endgroup$ – user251257 Jan 5 '18 at 2:45
  • $\begingroup$ In general, you need to check the second derivative to classify a local extremum. In this case, you can verify that the local extremum is a local minimum. $\endgroup$ – Math Lover Jan 5 '18 at 2:46
  • $\begingroup$ @АлександрБагдасарян The Prove is elementary in 2-d. You might want to try it. $\endgroup$ – user251257 Jan 5 '18 at 2:48
  • $\begingroup$ When you have a function which is the sum of some squares (like the one emerging from linear regression), you can show that they do not have a maximum. Wherever you are, there is some direction you can go to increase the functions value. This is not hard to prove, but Kahn should have mentioned this. However, since there are no maximums, he can be sure that zero derivative means minimum. $\endgroup$ – M. Winter Jan 5 '18 at 9:15
  • $\begingroup$ @M.Winter your explanation is the clearest one I've heard so far. $\endgroup$ – Alexander Bagdasaryan Jan 5 '18 at 15:03

That the critical point correspond to a mimimum squared error is a very intutively idea but it is not a simple fact to be proved in general.

In the simplest case the crucial fact is that we are dealing with the mimimization of the function of two variables:

$$e=f(m,b)=\sum (mx_i+b-y_i)^2$$

and at the critical point ($\nabla f=0$) we should also verify that the determinant of the Hessian matrix is positive

$$\begin{vmatrix} f_{mm}&f_{mb}\\f_{mb}&f_{bb} \end{vmatrix}=f_{mm}f_{bb}-(f_{mb})^2>0$$

It can be done by induction or by Cauchy-Schwartz inequality.

Here is a nice derivation A Quick Proof that the Least Squares Formulas Give a Local Minimum

  • $\begingroup$ I am curious why the answer was downvoted. $\endgroup$ – Alexander Bagdasaryan Jan 5 '18 at 14:11
  • $\begingroup$ @AlexanderBagdasaryan I didn't understand at first your question, then I've edited my answer! $\endgroup$ – gimusi Jan 5 '18 at 14:13
  • $\begingroup$ Even though your particular answer seems unaccessable to me, the file you've linked to has been extremely helpful, so I will accept your answer. $\endgroup$ – Alexander Bagdasaryan Jan 5 '18 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.