4
$\begingroup$

Let $Y_1, Y_2, \ldots, Y_n$ represent response variables and let $x_1, x_2,\ldots, x_n$ be the associated explanatory variables.

In the normal linear regression model, it's assumed that:

$$Y_i \sim N(\alpha + \beta x_i, \sigma^2).$$

The maximum likelihood estimate for $\beta$ is $\hat \beta = \frac{S_{XY}}{S_{XX}}$ where $S_{XY} = \sum_{i=1}^{n} (x_i - \bar x )(Y_i - \bar Y)$ and $S_{XX} = \sum_{i=1}^{n} (x_i - \bar x )^2$.

Clearly $\hat \beta$ is a normally distributed random variable (being a linear combination of normal random variables). I'm trying to show that it's variance is $\frac{\sigma^2}{S_{XX}}$ - but am really struggling.

I would really appreciate any pointers, hints, or solutions.

Thanks,

Jack

$\endgroup$

1 Answer 1

4
$\begingroup$

$\newcommand{\var}{\operatorname{var}}\newcommand{\cov}{\operatorname{cov}}\newcommand{\E}{\operatorname{E}}$ $$ \widehat\beta = \frac{\sum_{i=1}^n (x_i-\overline{x})(Y_i-\overline{Y})}{\sum_{i=1}^n (x_i - \overline x)^2} $$ $$ Y_i \sim N(\alpha + \beta x_i, \sigma^2) = N(\mu + \beta(x_i-\overline x), \sigma^2) $$ Observe that \begin{align} \cov(Y_i,\overline Y) & = \frac{\sigma^2} n, \\[10pt] \var(Y_i-\overline Y) & = \sigma^2 - 2\frac{\sigma^2} n + \frac{\sigma^2} n = \frac{n-1} n \sigma^2, \\[10pt] \cov(Y_i-\overline Y, Y_j - \overline Y) & = -\frac{\sigma^2} n \end{align} So \begin{align} & \var\left(\sum_{i=1}^n (x_i-\overline x)(Y_i - \overline Y) \right)\\[10pt] = {} & \sum_{i=1}^n \var\left( (x_i - \overline x) \var(Y_i - \overline Y) \right) + \sum_{i,j\,:\,i\, \ne \,j} \cov\left( (x_i-\overline x)(Y_i -\overline Y), (x_j-\overline x) (Y_j-\overline Y) \right) \\[10pt] = {} & \sum_{i=1}^n (x_i - \overline x)^2 \frac{(n-1)\sigma^2} n + 2\sum_{i,j\,:\,i\,<\,j} (x_i - \overline x)(x_j-\overline x) \left(\frac{-\sigma^2} n\right) \\[10pt] = {} & \frac{\sigma^2} n \left( (n-1)\sum_{i=1}^n (x_i - \overline x)^2 - \sum_{i,j\,:\,i\, \ne \, j} (x_i-\overline x)(x_j - \overline x) \right) \\[10pt] = {} & \frac{\sigma^2} n \left( n\sum_{i=1}^n (x_i-\overline x)^2 - \sum_{i,j} (x_i-\overline{x})(x_j-\overline x) \right) = \sigma^2 \sum_{i=1}^n (x_i-\overline x)^2. \end{align} Therefore $$ \var\left( \widehat\beta \right) = \frac{\sigma^2}{\sum_{i=1}^n(x_i-\overline x)^2} $$

If you like matrix algebra, then you can write $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} \sim N_n\left( \begin{bmatrix} 1 & x_1-\overline x \\ \vdots & \vdots \\ 1 & x_n - \overline{x} \end{bmatrix} \begin{bmatrix} \mu \\ \beta \end{bmatrix}, \sigma^2 \begin{bmatrix} 1 \\ & \ddots \\ & & 1 \end{bmatrix} \right) = N_n(X\gamma, \sigma^2 I_n). $$ So $$ \widehat\gamma = (X^\top X)^{-1} X^\top Y \qquad \text{(This is a $2\times 1$ matrix.)}, $$ so that \begin{align} \var\left( \,\widehat\gamma\,\right) & = (X^\top X)^{-1} X^\top\Big( \var(Y) \Big) X(X^\top X)^{-1} \\ & = (X^\top X)^{-1} X^\top\Big( \sigma^2 I_n \Big) X(X^\top X)^{-1} = \sigma^2 (X^\top X)^{-1}. \qquad \text{(This is a $2\times2$ matrix.)} \end{align} Then actually do the matrix inversion and get the same result we got above. This is less work than it would have been if I hadn't re-written this in terms of $\mu$ rather than $\alpha$, since I've made the matrix $X^\top X$ diagonal.

$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .