3
$\begingroup$

Let $Y_1, Y_2, \ldots, Y_n$ represent response variables and let $x_1, x_2,\ldots, x_n$ be the associated explanatory variables.

In the normal linear regression model, it's assumed that:

$$Y_i \sim N(\alpha + \beta x_i, \sigma^2).$$

The maximum likelihood estimate for $\beta$ is $\hat \beta = \frac{S_{XY}}{S_{XX}}$ where $S_{XY} = \sum_{i=1}^{n} (x_i - \bar x )(Y_i - \bar Y)$ and $S_{XX} = \sum_{i=1}^{n} (x_i - \bar x )^2$.

Clearly $\hat \beta$ is a normally distributed random variable (being a linear combination of normal random variables). I'm trying to show that it's variance is $\frac{\sigma^2}{S_{XX}}$ - but am really struggling.

I would really appreciate any pointers, hints, or solutions.

Thanks,

Jack

$\endgroup$
1
$\begingroup$

$\newcommand{\var}{\operatorname{var}}\newcommand{\cov}{\operatorname{cov}}\newcommand{\E}{\operatorname{E}}$ $$ \widehat\beta = \frac{\sum_{i=1}^n (x_i-\overline{x})(Y_i-\overline{Y})}{\sum_{i=1}^n (x_i - \overline x)^2} $$ $$ Y_i \sim N(\alpha + \beta x_i, \sigma^2) = N(\mu + \beta(x_i-\overline x), \sigma^2) $$ Observe that \begin{align} \cov(Y_i,\overline Y) & = \frac{\sigma^2} n, \\[10pt] \var(Y_i-\overline Y) & = \sigma^2 - 2\frac{\sigma^2} n + \frac{\sigma^2} n = \frac{n-1} n \sigma^2, \\[10pt] \cov(Y_i-\overline Y, Y_j - \overline Y) & = -\frac{\sigma^2} n \end{align} So \begin{align} & \var\left(\sum_{i=1}^n (x_i-\overline x)(Y_i - \overline Y) \right)\\[10pt] = {} & \sum_{i=1}^n \var\left( (x_i - \overline x) \var(Y_i - \overline Y) \right) + \sum_{i,j\,:\,i\, \ne \,j} \cov\left( (x_i-\overline x)(Y_i -\overline Y), (x_j-\overline x) (Y_j-\overline Y) \right) \\[10pt] = {} & \sum_{i=1}^n (x_i - \overline x)^2 \frac{(n-1)\sigma^2} n + 2\sum_{i,j\,:\,i\,<\,j} (x_i - \overline x)(x_j-\overline x) \left(\frac{-\sigma^2} n\right) \\[10pt] = {} & \frac{\sigma^2} n \left( (n-1)\sum_{i=1}^n (x_i - \overline x)^2 - \sum_{i,j\,:\,i\, \ne \, j} (x_i-\overline x)(x_j - \overline x) \right) \\[10pt] = {} & \frac{\sigma^2} n \left( n\sum_{i=1}^n (x_i-\overline x)^2 - \sum_{i,j} (x_i-\overline{x})(x_j-\overline x) \right) = \sigma^2 \sum_{i=1}^n (x_i-\overline x)^2. \end{align} Therefore $$ \var\left( \widehat\beta \right) = \frac{\sigma^2}{\sum_{i=1}^n(x_i-\overline x)^2} $$

If you like matrix algebra, then you can write $$ \begin{bmatrix} Y_1 \\ \vdots \\ Y_n \end{bmatrix} \sim N_n\left( \begin{bmatrix} 1 & x_1-\overline x \\ \vdots & \vdots \\ 1 & x_n - \overline{x} \end{bmatrix} \begin{bmatrix} \mu \\ \beta \end{bmatrix}, \sigma^2 \begin{bmatrix} 1 \\ & \ddots \\ & & 1 \end{bmatrix} \right) = N_n(X\gamma, \sigma^2 I_n). $$ So $$ \widehat\gamma = (X^\top X)^{-1} X^\top Y \qquad \text{(This is a $2\times 1$ matrix.)}, $$ so that \begin{align} \var\left( \,\widehat\gamma\,\right) & = (X^\top X)^{-1} X^\top\Big( \var(Y) \Big) X(X^\top X)^{-1} \\ & = (X^\top X)^{-1} X^\top\Big( \sigma^2 I_n \Big) X(X^\top X)^{-1} = \sigma^2 (X^\top X)^{-1}. \qquad \text{(This is a $2\times2$ matrix.)} \end{align} Then actually do the matrix inversion and get the same result we got above. This is less work than it would have been if I hadn't re-written this in terms of $\mu$ rather than $\alpha$, since I've made the matrix $X^\top X$ diagonal.

$\endgroup$
  • $\begingroup$ Thank you so much - and thanks for the bit on matrix algebra - a nice addition! $\endgroup$ – Jack Apr 24 '17 at 21:23
  • $\begingroup$ @Jack : I'm glad it helps. $\endgroup$ – Michael Hardy Apr 24 '17 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.