0
$\begingroup$

In machine learning, a simple linear regression model can be considered as follow:

hypothesis: $$h_{\theta}(x) = \theta_0+\theta_1x$$ and the cost function can be defined as:

$$J(\theta_0,\theta_1)=\frac{1}{2m}\sum_{i=1}^{m}(h_\theta(x^{(i)})-y^{(i)})^2$$

Then if we plot the cost function along with the two parameters, we would obtain a figure like this:enter image description here enter image description here (pictures are credited to Andrew Ng's machine learning course on Coursera)

My question is: Why would the figure be of many concentric ellipses when looking from the above? How to show this with rigorous mathematics?

$\endgroup$
2
  • $\begingroup$ The concentric ellipses represent isocontours, i.e. a constant value. So there are arbitrarily many ellipses for the arbitrarily many values between the minimum and value at the largest ellipse. Or did you mean why is the shape of the contour an ellipse in the first place? $\endgroup$
    – Benedict W. J. Irwin
    May 15, 2020 at 13:27
  • $\begingroup$ I mean why the shape of the contour is ellipse $\endgroup$
    – Fëanor Tang
    May 15, 2020 at 14:14

1 Answer 1

Reset to default
1
$\begingroup$

$$\sum(ax_i+b-y_i)^2=a^2\sum x_i^2+2ab\sum x_i+b^2\sum 1-2a\sum x_iy_i-2b\sum y_i+\sum y_i^2$$ which is a quadric in $a,b$.

The type of that quadric is governed by the sign of the discriminant

$$\sum x_i^2\sum 1-\left(\sum x_i\right)^2,$$ which is always positive (it is $\propto\sigma^2_x$). So any contour line is an ellipse.

$\endgroup$
4
  • $\begingroup$ Thank you for your suggestion. However, I don't quite understand whay the discriminant is of the form you provided, and what do you mean by $$\propto \sigma^2_x$$, how is it related to positive discriminant? $\endgroup$
    – Fëanor Tang
    May 15, 2020 at 14:13
  • $\begingroup$ @FëanorTang: just observe that the function is a sum of squares, hence it must describe a paraboloid. $\endgroup$
    – user65203
    May 15, 2020 at 14:27
  • $\begingroup$ could you please show how you get the expression for the determinant? and how does this related to being an ellipse? I think I have some knowledge gap here that need some prompts. Many thanks to you $\endgroup$
    – Fëanor Tang
    May 15, 2020 at 16:25
  • $\begingroup$ @FëanorTang: just observe that the function is a sum of squares (of linear terms), hence it must describe a paraboloid. $\endgroup$
    – user65203
    May 15, 2020 at 16:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .