Assuming a Gauss-Markov model such that $H_0 : C\beta = d$, (I'm posting here as suggested by the SE Statistics)

(a) how do I prove that the variance of $C\hat\beta \sim N(C\beta, \sigma^2C(X'X)^{-1}C')$?

(b) AND then, what if $C\beta$ is NOT estimable. What is the simplified expression for the Variance $\operatorname{Var}(C(X'X)^{-1}X'y)$ if not??

My Work...Which I Know is Not Correct, when assuming it is testable:

Prove $\operatorname{Var}(C\hat\beta) = \sigma^2C(X'X)^{-1}C'$.

Let $\hat\beta = (X'X)^{-1}X'y$ By definition.

$C = AX$ Given $C\beta$ is estimable, for some matrix $A$.

$=\sigma^2AX(X'X)^{-1}AX'y$ Substitution.


$\operatorname{Var}(y)$ in linear transformation $= A\operatorname{Var}(y)A'$.

$Px=X(X'X)^{-1}X'$ Def. of projection operator.

$=\sigma^2APxAA'$ Substitution.

....don't know how to finish.

  • $\begingroup$ It's been a while since I've thought about estimability. Probably I'll look at this tomorrow${}\,\ldots\qquad$ $\endgroup$ – Michael Hardy Jan 20 '16 at 6:01
  • $\begingroup$ I changed $(X'X)^-$ to $(X'X)^{-1}$, but now I'm wondering if you intended the notation $A^-$ to mean a generalized inverse of a matrix that is not of full rank? $\qquad$ $\endgroup$ – Michael Hardy Jan 21 '16 at 0:33
  • $\begingroup$ Your answer below is what I was looking for. I meant that the rank$(X)=p$, i.e. full column rank. Should I state this near the top of the proof? $\endgroup$ – Jennifer Cooke Jan 22 '16 at 4:39

The design matrix $X \in\mathbb R^{n\times p}$ typically has many more rows than columns. If the columns of $X$ are linearly independent, then $X'X$ is invertible. In that case we have $$ \hat \beta = (X'X)^{-1} X' y. $$ Therefore \begin{align} \operatorname{var}(\hat\beta) & = \Big( (X'X)^{-1} X'\Big)\Big( \operatorname{var}(y)\Big)\Big( (X'X)^{-1} X'\Big)' \\[8pt] & = \Big( (X'X)^{-1} X'\Big)\Big( \sigma^2 I_n \Big)\Big( (X'X)^{-1} X'\Big)' \\[8pt] & = \sigma^2 \Big( (X'X)^{-1} X' \Big) \Big( X'(X'X)^{-1}\Big) = \sigma^2 (X'X)^{-1}. \end{align} Hence \begin{align} \operatorname{var}(C\hat\beta) = C\Big( \operatorname{var}(\hat\beta)\Big) C'. \end{align} The expected value is simpler: $$ \operatorname{E}(C\hat\beta) = C \operatorname{E}(\hat\beta). $$ Generally if $W\in\mathbb R^k$ is normally distributed and $M\in\mathbb R^{\ell\times k}$ is a constant (i.e. non-random) matrix then $MW\in\mathbb R^\ell$ is normally distributed.

No linear combination of components of $\beta$ can fail to be estimable unless the columns of $X$ are linearly dependent, and that's a separate case I will deal with here later${}\,\ldots$

| cite | improve this answer | |
  • $\begingroup$ thank you so much for not only answering, but also for mentioning and teaching in the answer. For example I didn't think about how the columns have to be linearly independent to be invertible. $\endgroup$ – Jennifer Cooke Jan 22 '16 at 5:01
  • $\begingroup$ Please review your answer, I looked back and isn't $Var\hat\beta$ = A times Var(y) times A', and instead of $A'$ you wrote $A^-1$? So on the last line, we distribute the transpose, not the inverse, in order to get C'. $\endgroup$ – Jennifer Cooke Jan 23 '16 at 18:05
  • $\begingroup$ @JenniferCooke : That's a pretty bad typo. I've (I hope) fixed it. $\qquad$ $\endgroup$ – Michael Hardy Jan 23 '16 at 21:10
  • $\begingroup$ I have a question I posted a about an hour ago on estimability based off a design matrix I'm supposed to know how to pick. Do you have time for a quick chat on this? stats.stackexchange.com/q/192139/100762 $\endgroup$ – Jennifer Cooke Jan 23 '16 at 23:32
  • $\begingroup$ Maybe I'll look at that this afternoon. (I think this is the first time I've seen someone on stackexchange write "based off" rather than "based on". Quite a strange alien locution to me, apparently prevailing among the very young recently, although about a month ago I actually heard an adult say it.) $\endgroup$ – Michael Hardy Jan 24 '16 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.