# Variance of Beta in the Normal Linear Regression Model

Let $Y_1, Y_2, \ldots, Y_n$ represent response variables and let $x_1, x_2,\ldots, x_n$ be the associated explanatory variables.

In the normal linear regression model, it's assumed that:

$$Y_i \sim N(\alpha + \beta x_i, \sigma^2).$$

The maximum likelihood estimate for $\beta$ is $\hat \beta = \frac{S_{XY}}{S_{XX}}$ where $S_{XY} = \sum_{i=1}^{n} (x_i - \bar x )(Y_i - \bar Y)$ and $S_{XX} = \sum_{i=1}^{n} (x_i - \bar x )^2$.

Clearly $\hat \beta$ is a normally distributed random variable (being a linear combination of normal random variables). I'm trying to show that it's variance is $\frac{\sigma^2}{S_{XX}}$ - but am really struggling.

I would really appreciate any pointers, hints, or solutions.

Thanks,

Jack

If you like matrix algebra, then you can write $$\begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} \sim N_n\left( \begin{bmatrix} 1 & x_1-\overline x \\ \vdots & \vdots \\ 1 & x_n - \overline{x} \end{bmatrix} \begin{bmatrix} \mu \\ \beta \end{bmatrix}, \sigma^2 \begin{bmatrix} 1 \\ & \ddots \\ & & 1 \end{bmatrix} \right) = N_n(X\gamma, \sigma^2 I_n).$$ So $$\widehat\gamma = (X^\top X)^{-1} X^\top Y \qquad \text{(This is a 2\times 1 matrix.)},$$ so that \begin{align} \var\left( \,\widehat\gamma\,\right) & = (X^\top X)^{-1} X^\top\Big( \var(Y) \Big) X(X^\top X)^{-1} \\ & = (X^\top X)^{-1} X^\top\Big( \sigma^2 I_n \Big) X(X^\top X)^{-1} = \sigma^2 (X^\top X)^{-1}. \qquad \text{(This is a $2\times2$ matrix.)} \end{align} Then actually do the matrix inversion and get the same result we got above. This is less work than it would have been if I hadn't re-written this in terms of $\mu$ rather than $\alpha$, since I've made the matrix $X^\top X$ diagonal.