# How to Prove Variance using Matrices $\operatorname{Var}(C\hat\beta) = \sigma^2C(X'X)^-C'$?

Assuming a Gauss-Markov model such that $H_0 : C\beta = d$, (I'm posting here as suggested by the SE Statistics)

(a) how do I prove that the variance of $C\hat\beta \sim N(C\beta, \sigma^2C(X'X)^{-1}C')$?

(b) AND then, what if $C\beta$ is NOT estimable. What is the simplified expression for the Variance $\operatorname{Var}(C(X'X)^{-1}X'y)$ if not??

My Work...Which I Know is Not Correct, when assuming it is testable:

Prove $\operatorname{Var}(C\hat\beta) = \sigma^2C(X'X)^{-1}C'$.

Let $\hat\beta = (X'X)^{-1}X'y$ By definition.

$C = AX$ Given $C\beta$ is estimable, for some matrix $A$.

$=\sigma^2AX(X'X)^{-1}AX'y$ Substitution.

$=\sigma^2AX(X'X)^{-1}X'AA'$

$\operatorname{Var}(y)$ in linear transformation $= A\operatorname{Var}(y)A'$.

$Px=X(X'X)^{-1}X'$ Def. of projection operator.

$=\sigma^2APxAA'$ Substitution.

....don't know how to finish.

• It's been a while since I've thought about estimability. Probably I'll look at this tomorrow${}\,\ldots\qquad$ – Michael Hardy Jan 20 '16 at 6:01
• I changed $(X'X)^-$ to $(X'X)^{-1}$, but now I'm wondering if you intended the notation $A^-$ to mean a generalized inverse of a matrix that is not of full rank? $\qquad$ – Michael Hardy Jan 21 '16 at 0:33
• Your answer below is what I was looking for. I meant that the rank$(X)=p$, i.e. full column rank. Should I state this near the top of the proof? – Jennifer Cooke Jan 22 '16 at 4:39

The design matrix $X \in\mathbb R^{n\times p}$ typically has many more rows than columns. If the columns of $X$ are linearly independent, then $X'X$ is invertible. In that case we have $$\hat \beta = (X'X)^{-1} X' y.$$ Therefore \begin{align} \operatorname{var}(\hat\beta) & = \Big( (X'X)^{-1} X'\Big)\Big( \operatorname{var}(y)\Big)\Big( (X'X)^{-1} X'\Big)' \\[8pt] & = \Big( (X'X)^{-1} X'\Big)\Big( \sigma^2 I_n \Big)\Big( (X'X)^{-1} X'\Big)' \\[8pt] & = \sigma^2 \Big( (X'X)^{-1} X' \Big) \Big( X'(X'X)^{-1}\Big) = \sigma^2 (X'X)^{-1}. \end{align} Hence \begin{align} \operatorname{var}(C\hat\beta) = C\Big( \operatorname{var}(\hat\beta)\Big) C'. \end{align} The expected value is simpler: $$\operatorname{E}(C\hat\beta) = C \operatorname{E}(\hat\beta).$$ Generally if $W\in\mathbb R^k$ is normally distributed and $M\in\mathbb R^{\ell\times k}$ is a constant (i.e. non-random) matrix then $MW\in\mathbb R^\ell$ is normally distributed.
No linear combination of components of $\beta$ can fail to be estimable unless the columns of $X$ are linearly dependent, and that's a separate case I will deal with here later${}\,\ldots$
• Please review your answer, I looked back and isn't $Var\hat\beta$ = A times Var(y) times A', and instead of $A'$ you wrote $A^-1$? So on the last line, we distribute the transpose, not the inverse, in order to get C'. – Jennifer Cooke Jan 23 '16 at 18:05
• @JenniferCooke : That's a pretty bad typo. I've (I hope) fixed it. $\qquad$ – Michael Hardy Jan 23 '16 at 21:10