2
$\begingroup$

For context,
the linear model is
$$\mathbf{y} = X\mathbf{\beta} + \mathbf{\varepsilon} $$ where $\mathbf{y}$ is the vector of responses $y_i,i=1,2,\ldots,n$, $X$ is the deterministic design matrix $n\times p$ and $\beta$ is the vector of parameters, $p\times 1$.
Also, $\varepsilon$ is the vector of uncorrelated errors with zero mean, $\varepsilon_i,i=1,2,\ldots,n$ where $\mathrm{Var}(\varepsilon_i) = \sigma^2.$

Now, the MLE of $\sigma^2$ in the multiple linear model is
$$\hat{\sigma}^2 := \frac{1}{n} (y-XB)^T (y-XB) $$
And I'm given the formula that for any random $k\times 1$ vector $u$, and constant $k\times k$ matrix $A$, we have

$$\mathbb{E}(u^T A u) = \mathrm{tr}(AV) + \mu^T A \mu$$ where $V=\mathrm{Var}(u), \ \mu = \mathbb{E}(u)$.

So I try to apply this to my MLE.
$$\hat{\sigma}^2 := \frac{1}{n} \varepsilon^T \varepsilon \\ \mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} \mathbb{E}(\varepsilon ^T I_{n\times n} \varepsilon) $$
with $A = I_{n\times n}, u = \varepsilon$.
So $V = \mathrm{Var}(\varepsilon) = \sigma^2 I_{n\times n}, \mu = \mathbb{E}(\varepsilon) = \mathbf{0}$.
hence, $$\mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} ( \mathrm{tr}(I_{n\times n}\sigma^2 I_{n\times n}) = \sigma^2.$$

However, this is incorrect as the solution says that it is $\frac{n-p}{n}\sigma^2$ and it also shows me how it's done. However, I'm not sure where I went wrong.

$\endgroup$
0
$\begingroup$

The problem here is that even tough the error term $\epsilon$ has variance given by $\sigma^2 I_{n\times n}$ the residuals $\hat{\epsilon}$ have a different variance.

Note that, by definition

$\hat{\sigma}^2=\frac{1}{n}(Y-X\hat\beta)'(Y-X\hat\beta)=\frac{1}{n}\hat{\epsilon}'\hat{\epsilon}$

You should check that, $\hat\epsilon=(I_{n\times n}-P)Y$ where $P$ is the projection matrix $P:=X(X'X)^{-1}X'$. Hence, $Var(\hat\epsilon)=\sigma^2(I_{n\times n}-P)(I_{n\times n}-P)'=(I_{n\times n}-P)$ since $(I_{n\times n}-P)$ is idempotent.

Now, use this as $V$ in your procedure and that should give you the correct result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.