1
$\begingroup$

Assume a linear regression model $y=X\beta + \epsilon$ with $\epsilon \sim N(0,\sigma^2I)$ and $\hat y=Xb$ where $b=(X'X)^{-1}X'y$. Besides $H=X(X'X)^{-1}X'$ is the linear projection from the response space to the span of $X$, i.e., $\hat y=Hy$.

Now I want to calculate $Var[b]$ but what I get is an $k\times k$ matrix, not an $n \times n$ one. Here's my calculation:

\begin{align} Var[b] =&\; Var[(X'X)^{-1}X'y]\\ =& \;(X'X)^{-1}X'\,\underbrace{Var[y]}_{= \sigma^2I}X(X'X)^{-1}\\\\ \text{Here you can }& \text{see already this thing will be k $\times$ k} \\\\ =&\; \sigma^2 \underbrace{(X'X)^{-1}X'X}_{I}(X'X)^{-1}\\ =& \sigma^2(X'X)^{-1}\, \in R^{k\times k} \end{align}

What am I doing wrong?

Besides, are $E[b]=\beta$, $E[\hat y]=HX\beta$, $Var[\hat y]=\sigma^2H$, $E[y-\hat y]=(I-H)X\beta$, $Var[y-\hat y]=(I-H)\sigma^2$ correct (this is just on a side note, my main question is the one above)?

$\endgroup$
1

1 Answer 1

1
$\begingroup$

The covariance matrix for $b$ (the estimator for $\beta$) should be $k\times k$. If the $X$ matrix is $n\times k$ then $\beta$ has to be $k\times 1$; otherwise the product $X\beta$ wouldn't be $n\times 1$.

So if $\beta$ is a constant vector of $k$ parameters, then its estimator $b$ is a random vector with $k$ elements. Therefore the covariance matrix for $b$ consists of covariances for all possible combinations of two members selected from the random vector, hence it must be a $k\times k$ matrix.

To answer your side notes, all your calculations are correct but some can be simplified further. Check that $HX=X$, so that $E[\hat y]=X\beta$, and $E[y-\hat y]=0$.

$\endgroup$
1
  • $\begingroup$ Yes, your dimensions are right, I didn't explicitly write them down, sorry. I don't quite understand why it's $k \times k$, because I thought it's the covariance between data points. $\endgroup$ Dec 1, 2020 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.