# shape of contour plots in machine learning problems

In machine learning, a simple linear regression model can be considered as follow:

hypothesis: $$h_{\theta}(x) = \theta_0+\theta_1x$$ and the cost function can be defined as:

$$J(\theta_0,\theta_1)=\frac{1}{2m}\sum_{i=1}^{m}(h_\theta(x^{(i)})-y^{(i)})^2$$

Then if we plot the cost function along with the two parameters, we would obtain a figure like this:  (pictures are credited to Andrew Ng's machine learning course on Coursera)

My question is: Why would the figure be of many concentric ellipses when looking from the above? How to show this with rigorous mathematics?

• The concentric ellipses represent isocontours, i.e. a constant value. So there are arbitrarily many ellipses for the arbitrarily many values between the minimum and value at the largest ellipse. Or did you mean why is the shape of the contour an ellipse in the first place? – Benedict W. J. Irwin May 15 '20 at 13:27
• I mean why the shape of the contour is ellipse – Fëanor Tang May 15 '20 at 14:14

$$\sum(ax_i+b-y_i)^2=a^2\sum x_i^2+2ab\sum x_i+b^2\sum 1-2a\sum x_iy_i-2b\sum y_i+\sum y_i^2$$ which is a quadric in $$a,b$$.
$$\sum x_i^2\sum 1-\left(\sum x_i\right)^2,$$ which is always positive (it is $$\propto\sigma^2_x$$). So any contour line is an ellipse.
• Thank you for your suggestion. However, I don't quite understand whay the discriminant is of the form you provided, and what do you mean by $$\propto \sigma^2_x$$, how is it related to positive discriminant? – Fëanor Tang May 15 '20 at 14:13