# Least Squares Fit

So here's the question:

Given a collection of points $(x_1,y_1), (x_2,y_2),\ldots,(x_n,y_n)$, let $x=(x_1,x_2,\ldots,x_n)^T$, $y=(y_1,y_2,\ldots,y_n)^T$, $\bar{x}=\frac{1}{n} \sum\limits_{i=1}^n x_i$, $\bar{y}=\frac{1}{n} \sum\limits_{i=1}^n y_i$.
Let $y=c_0+c_1x$ be the linear function that gives the best least squares fit to the points. Show that if $\bar{x}=0$, then $c_0=\bar{y}$ and $c_1=\frac{x^Ty}{x^Tx}$.

I've managed to do all the problems in this least squares chapter but this one has me completely and utterly stumped. I'm not entirely sure what the question is even telling me in terms of information nor do I get what it's asking. Any ideas on where to start?

-
Start by representing mathematically what it means that this linear function is the best least squares fit. Best in what sense? what is it minimizing? etc... – Bitwise Oct 14 '12 at 3:40
@Bitwise is it the error? – Charlie Yabben Oct 14 '12 at 3:51
yes, the regression line is minimizing the squared error right? meaning that for each data point, there is a matching point on the line, and the squared distance between them should be minimal. Try writing that down explicitly. – Bitwise Oct 14 '12 at 3:59
@Bitwise okay I understand but i'm not really following on how to write that down explicitly – Charlie Yabben Oct 14 '12 at 4:07
@CharlieYabben You sure the formula for $\hat c_1$ in the question is correct? – user60610 Apr 1 '13 at 19:15

You want to minimize:

$||\hat{y}-y||^2 = ||(c_0+c_1x)-y||^2$

where $\hat{y}$ are given by the linear regression. Intuitively, the best line minimizes the squared error between the "observed" ys and the "predicted" ys.

So, you need to find that $c_0,c_1$ that minimize this expression and show they are equal to those specified above.

Can you go on from there?

-
is $\hat{y}$ the same thing as $\bar{y}$? If so I would plug in $0$ for $x$ and that would leave me with $||\hat{y}-y||^2=||c_0-y||^2$. Then I sqrt root both sides and have $\hat{y}-y=c_0-y$ which then leads me to $\hat{y}=c_0$. Therefore I just proved one of them. Is that right? – Charlie Yabben Oct 14 '12 at 16:48
Unfortunately, no. $\hat{y}$ and $y$ are vectors, while $\bar{y}$ is a scalar (number) which is actually just the mean of the vector $y$. $\hat{y}$ is the vector created by taking the vector $x$ and multiplying each coordinate by a number ($c_1$) and then adding to each coordinate a number ($c_0$). In other words, these are the values "predicted" by the regression line for $x$. – Bitwise Oct 14 '12 at 17:05

The OLS estimators for $c_0$ and $c_1$ under the general setting(with no $\bar x=0$) are: $$c_0=\bar y- c_1\bar x, c_1= \frac{(x-\bar x\boldsymbol{1})^T(y-\bar y\boldsymbol{1})}{(x-\bar x\boldsymbol{1})^T(x-\bar x\boldsymbol{1})},$$ where $\boldsymbol{1}=(1,...,1)^T$ is a vector of one's and of length $n$. With the additional assumption $\bar x=0$, this reduces to $$c_0=\bar y, c_1= \frac{x^T(y-\bar y\boldsymbol{1})}{x^Tx}$$

-