0
$\begingroup$

In machine learning, a simple linear regression model can be considered as follow:

hypothesis: $$h_{\theta}(x) = \theta_0+\theta_1x$$ and the cost function can be defined as:

$$J(\theta_0,\theta_1)=\frac{1}{2m}\sum_{i=1}^{m}(h_\theta(x^{(i)})-y^{(i)})^2$$

Then if we plot the cost function along with the two parameters, we would obtain a figure like this:enter image description here enter image description here (pictures are credited to Andrew Ng's machine learning course on Coursera)

My question is: Why would the figure be of many concentric ellipses when looking from the above? How to show this with rigorous mathematics?

$\endgroup$
  • $\begingroup$ The concentric ellipses represent isocontours, i.e. a constant value. So there are arbitrarily many ellipses for the arbitrarily many values between the minimum and value at the largest ellipse. Or did you mean why is the shape of the contour an ellipse in the first place? $\endgroup$ – Benedict W. J. Irwin May 15 '20 at 13:27
  • $\begingroup$ I mean why the shape of the contour is ellipse $\endgroup$ – Fëanor Tang May 15 '20 at 14:14
1
$\begingroup$

$$\sum(ax_i+b-y_i)^2=a^2\sum x_i^2+2ab\sum x_i+b^2\sum 1-2a\sum x_iy_i-2b\sum y_i+\sum y_i^2$$ which is a quadric in $a,b$.

The type of that quadric is governed by the sign of the discriminant

$$\sum x_i^2\sum 1-\left(\sum x_i\right)^2,$$ which is always positive (it is $\propto\sigma^2_x$). So any contour line is an ellipse.

$\endgroup$
  • $\begingroup$ Thank you for your suggestion. However, I don't quite understand whay the discriminant is of the form you provided, and what do you mean by $$\propto \sigma^2_x$$, how is it related to positive discriminant? $\endgroup$ – Fëanor Tang May 15 '20 at 14:13
  • $\begingroup$ @FëanorTang: just observe that the function is a sum of squares, hence it must describe a paraboloid. $\endgroup$ – Yves Daoust May 15 '20 at 14:27
  • $\begingroup$ could you please show how you get the expression for the determinant? and how does this related to being an ellipse? I think I have some knowledge gap here that need some prompts. Many thanks to you $\endgroup$ – Fëanor Tang May 15 '20 at 16:25
  • $\begingroup$ @FëanorTang: just observe that the function is a sum of squares (of linear terms), hence it must describe a paraboloid. $\endgroup$ – Yves Daoust May 15 '20 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.