My question I am trying to attempt is below as follows:
Consider a data set ($x_1,y_1$), ($x_2, y_2$),..., ($x_n,y_n$), for which we want to find the line of best fit. The "method of least-squares assume the line of best fit has the form $y = ax +b$, and minimize the square of the error, defined as $$E(a,b) = \frac 1n \sum_i^n(y_i-y(x_i))^2$$
1. Let the mean of a sequence be $\bar z = \frac 1n \sum^n_i_=_1 z_i$, hence show that $$E(a,b) = \bar y^2 - 2a \overline x \overline y - 2b \bar y + a^2 \bar x^2 + 2ab \bar x + b^2$$ 2. Show that the only critical point of E(a,b) is $$a = \frac {\bar x . \bar y - \overline x\overline y}{(\bar x)^2 - \bar x^2}, b = \frac {-\bar x^2 . \bar y + \bar x .\overline x\overline y}{(\bar x)^2 - \bar x^2}$$ 3. Given that $\bar x^2 > (\bar x)^2$, show that the critical point is a local minimum. 4. The following table shows the world population y (in billions) for five different years.
Let x = 3 represent the year 2003. $$ \begin{array}{c|c|c|c|c} Year,2000 + 3 & 2003 & 2005 & 2007 & 2009 & 2011 \\ \hline Population,y & 6.3 & 6.5 & 6.6 & 6.8 & 6.9\\ \end{array} $$ Use the method of least-squares to calculate a and b, and predict the population for the year 2020.

Can somebody please give me pointers on how to do each of the questions? I'm not quite sure where to begin with each question, I have a vague idea for question 2 but the rest of them I'm not sure.


OK, just to give you some intuition. For problem no. 1, expand the expression under the square:

$$\sum_{i=1}^{n}\left(y_{i}-y(x_{i})\right)^{n}=\sum_{i=1}^{n}\left(y_{i}-ax_{i}-b\right)^{2}=\sum_{i=1}^{n}\left(b^{2}+2abx_{i}+a^{2}x_{i}^{2}-2by_{i}-2ax_{i}y_{i}+y^{2}\right)$$ Hence: $$E(a,b)=\frac{1}{n}\sum_{i=1}^{n}b^{2}+\frac{1}{n}\sum_{i=1}^{n}2abx_{i}+\frac{1}{n}\sum_{i=1}^{n}a^{2}x_{i}^{2}-\frac{1}{n}\sum_{i=1}^{n}2by_{i}-\frac{1}{n}\sum_{i=1}^{n}2ax_{i}y_{i}+\frac{1}{n}\sum_{i=1}^{n}y^{2}$$ Now, use the definition of mean given in the problem, use the fact that you can bring a constant in front of the series sign, and finally, use the fact that if $\alpha$ is a constant, $\sum_{i=1}^{n}\alpha=n\cdot\alpha$


In order to solve problem no 2., you have to find the partial derivatives of $E(a,b)$ with respect to $a$ and $b$ and set them to zero:

$$\frac{\partial{}E(a,b)}{\partial{}a}=-2\bar{x}\bar{y}+2a\bar{x}^{2}+2b\bar{x}=0$$ and $$\frac{\partial{}E(a,b)}{\partial{}b}=-2\bar{y}+2a\bar{x}+2b=0$$ Solve the above system of equations for $a$ and $b$, and check with the answer provided.


Divide the first equation by $2$. You get $-\bar{x}\bar{y}+a\bar{x}^{2}+b\bar{x}=0$. From the second equation, as you wrote, you obtained $a=\left(\bar{y}-b\right)/\bar{x}$. Now plug it into that first equation. You get: $$-\bar{x}\bar{y}+\frac{\bar{y}-b}{\bar{x}}\cdot\bar{x}^{2}+b\bar{x}=0$$ which is: $$\frac{\bar{y}-b}{\bar{x}}\cdot\bar{x}^{2}+b\bar{x}=\bar{x}\bar{y}$$ Now multiply both sides by $\bar{x}$: $$\bar{y}\bar{x}^{2}-b\bar{x}^{2}+b\left(\bar{x}\right)^{2}=\bar{x}\cdot\bar{x}\bar{y}$$ Reorganizing the terms yields: $$b\left(\bar{x}\right)^{2}-b\bar{x}^{2}=\bar{x}\cdot\bar{x}\bar{y}-\bar{x}^{2}\bar{y}$$ Solving for $b$: $$b=\frac{\bar{x}\cdot\bar{x}\bar{y}-\bar{x}^{2}\bar{y}}{\left(\bar{x}\right)^{2}-\bar{x}^{2}}.$$ As you can see, the result is correct.

  • $\begingroup$ I've done question 1 but now I'm stuck on question 2, how do I do that? $\endgroup$ – user147074 May 1 '14 at 14:20
  • $\begingroup$ Check the edit. $\endgroup$ – Johnny Westerling May 1 '14 at 14:28
  • $\begingroup$ Yes, I follow you, I got to that point before where $a = \frac {y-b}{x}$ and $b = y-ax$, I'm not sure what to do with them now, do I substitute a and b back into $E(a,b)$? Thanks for your patience. $\endgroup$ – user147074 May 1 '14 at 14:32
  • $\begingroup$ Now try solving for $a$. This is seriously nothing more than a linear system of two equations. You should be able to cope with that :-) $\endgroup$ – Johnny Westerling May 1 '14 at 14:54
  • $\begingroup$ Ok I've now done 1 and 2, with 3, I'm not sure how to approach this, thanks for your help so far $\endgroup$ – user147074 May 1 '14 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.