# properties of least square estimators in regression

$Y_i=\beta_0+\beta_1 X_i+\epsilon_i$ where $\epsilon_i$ is normally distributed with mean $0$ and variance $\sigma^2$ .
The least square estimators of this model are $\hat\beta_0$ and $\hat\beta_1$.
How can I show that $\hat\beta_0$ and $\hat\beta_1$ are linear functions of $y_i$?
Also it says that both estimators are normally distributed.How come they normally distributed?I know that linear functions of normally distributed variables are also normally distributed. . please explain this to me

• Do you mean $\beta_1 X_i$ instead of $\beta_1 + X_i$? – Petite Etincelle Apr 19 '14 at 14:36
• Yes corrected that one.It was a mistake – clarkson Apr 19 '14 at 15:29

One has $$\hat\beta_1 = \frac{\sum_{i=1}^n (y_i-\bar y)(x_i-\bar x)}{\sum_{i=1}^n (x_i - \bar x)^2}$$ where $\bar y = (y_1+\cdots+y_n)/n$ and $\bar x = (x_1+\cdots+x_n)/n$.

This is nonlinear as a function of $x_1,\ldots,x_n$ since there is division by a function of the $x$s and there is squaring. But it is linear as a function of $y_1,\ldots,y_n$. To see that, first observe that the denominator does not depend on $y_1,\ldots,y_n$, so we need only look at the numerator.

So look at $$y_i-\bar y = y_i - \frac{y_1 + \cdots + y_i + \cdots + y_n}{n} = \frac{-y_1 - y_2 - \cdots+(n-1)y_i-\cdots - y_n}{n}$$ This is linear in $y_1,\ldots,y_n$. Since the quantities $x_i-\bar x$, $i=1,\ldots,n$ do not depend on $y_1,\ldots,y_n$, the expression $$\sum_{i=1}^n (y_i-\bar y)(x_i-\bar x)$$ is a linear combination of expressions each of which we just said is linear in $y_1,\ldots,y_n$. It is therefore itself a linear combination of $y_1,\ldots,y_n$.

Next, we have $\bar y = \hat\beta_0 + \hat\beta_1 \bar x$, so $\beta_0 = \bar y - \hat\beta_1\bar x$. Since $\hat y$ is a linear combination of $y_1,\ldots,y_n$ and we just got done showing that $\hat\beta_1$ is a linear combination of $y_1,\ldots,y_n$, and $\bar x$ does not depend on $y_1,\ldots,y_n$, it follows that $\hat\beta_0$ is a linear combination of $y_1,\ldots,y_n$.

MATRIX VERSION:

$$\begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} = \begin{bmatrix} 1 & X_1 \\ \vdots & \vdots \\ 1 & X_n \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \vdots \\ \varepsilon_n \end{bmatrix}$$ $$Y = M\beta + \varepsilon$$ $$\varepsilon \sim N_n( 0_n, \sigma^2 I_n)$$ where $0_n\in\mathbb R^{n\times 1}$ and $I_n\in\mathbb R^{n\times n}$ is the identity matrix. Consequently $$Y\sim N_n(M\beta,\sigma^2 I_n).$$

One can show (and I show further down below) that $$\hat\beta = (M^\top M)^{-1}M^\top Y. \tag 1$$ Therefore $$\hat\beta \sim N_2(\Big((M^\top M)^{-1}M^\top\Big) M\beta,\quad (M^\top M)^{-1}M^\top\Big(\sigma^2 I_n\Big)M(M^\top M)^{-1})$$

$$= N_2( M\beta,\quad \sigma^2 (M^\top M)^{-1}).$$

Here I have used the fact that when one multiplies a normally distributed column vector on the left by a constant (i.e. non-random) matrix, the expected value gets multiplied by the same matrix on the left and the variance gets multiplied on the left by that matrix and on the right by its transpose.

So how does one prove $(1)$?

"Least squares" means the vector $\hat Y$ of fitted values is the orthogonal projection of $Y$ onto the column space of $M$. That projection is $$\hat Y = M(M^\top M)^{-1}M^\top Y. \tag 2$$ To see that that is the orthogonal projection, consider two things: Suppose $Y$ were orthogonal to the column spacee of $M$. Then the product $(2)$ must be $0$ since the product of the last two factors, ,$M^\top Y$, would be $0$. The suppose $Y$ is actually in the column space of $M$. Then $Y=M\gamma$ for some $\gamma\in \mathbb R^{2\times 1}$. Put $M\gamma$ into $(2)$ and simplify and the product will be $M\gamma=Y$, so that vectors in the column space are mapped to themselves.

Now we have $$M\hat\beta=\hat Y = M(M^\top M)^{-1} M^\top Y. \tag 3$$ If we could multiply both sides of $(3)$ on the left by an inverse of $M$, we'd get $(1)$. But $M$ is not a square matrix and so has no inverse. But $M$ is a matrix with linearly independent columns and therefore has a left inverse, and that does the job. Its left inverse is $$(M^\top M)^{-1}M^\top.$$ The left inverse is not unique, but this is the one that people use in this context.

• I don't know the matrix form.Can you please explain it in another way – clarkson Apr 19 '14 at 15:47

As a complement to the answer given by @MichaelHardy, substituting $Y = M\beta + \varepsilon$ (i.e., the regression model) in the expression of the least squares estimator may be helpful to see why the OLS estimator is normally distributed.

$$\begin{eqnarray} \begin{array}{l} \hat\beta &=& (M^\top M)^{-1}M^\top \underbrace{Y}_{Y = M\beta + \varepsilon} \\ \hat\beta &=& (M^\top M)^{-1} (M^\top M)\beta + (M^\top M)^{-1}M^\top \varepsilon . \\ \end{array} \end{eqnarray}$$

$$\hat\beta = \beta + (M^\top M)^{-1}M^\top \varepsilon . \tag 1$$

$\beta$ is a constant vector (the true and unknown values of the parameters). Also, under the assumptions of the classical linear regression model the regressor variables arranged by columns in $M$ are fixed (non-stochastic) and the error term $\varepsilon$ is distributed normally distributed with mean zero and variance $\sigma^2$, $\epsilon_t \sim NID(0, \sigma^2)$.

$\hat\beta$ is a linear function of a normally distributed variable and, hence, $\hat\beta$ is also normal. In particular, as mentioned in another answer, $\hat\beta \sim N(\beta, \sigma^2(M^\top M)^{-1})$, which is straightforward to check from equation (1):

$$E(\hat\beta) = E\left( \beta + (M^\top M)^{-1}M^\top \varepsilon \right) = \beta + (M^\top M)^{-1}M^\top \underbrace{E\left(\varepsilon \right)}_{0} = \beta$$

$$\begin{eqnarray} \begin{array}{l} \hbox{Var}(\hat\beta) &=& E\left( [\hat\beta - E(\hat\beta)] [\hat\beta - E(\hat\beta)]^\top\right) = E\left( (M^\top M)^{-1}M^\top \varepsilon\varepsilon^\top M(M^\top M)^{-1} \right) \\ &=& (M^\top M)^{-1}M^\top \underbrace{E\left( \varepsilon\varepsilon^\top \right)}_{\sigma^2} M(M^\top M)^{-1} = \sigma^2 (M^\top M)^{-1} . \end{array} \end{eqnarray}$$

The above calculations make use of the definition of the error term, $NID(0, \sigma^2)$, and the fact that the regressors $M$ are fixed values. The first result $\hat\beta=\beta$ implies that the OLS estimator is unbiased.