How to prove that $\hat\sigma^2$ has $\chi^2_{n-p}$ distribution (linear regression)

Question

Consider the linear regression model:

$$Y_i=r(x_i)+\varepsilon_i\equiv\sum_{j = 1}^p x_{ij} \beta _j + \varepsilon _i,\quad i=1,\ldots,n.$$

where $x_1,\ldots,x_n\in \mathbb{R}^p$ are fixed, $E(\varepsilon_i)=0$, $\operatorname{Var}(\varepsilon_i)=\sigma^2$. Denote $Y=(Y_1,\ldots,Y_n)^T$, $\beta=(\beta_1,\ldots,\beta_p)^T$, $X=(x_{ij})_{n\times p}$. As is known, $\hat \beta = \arg\min\limits_{\beta \in \mathbb{R}^p} (Y - X\beta)^T (Y - X\beta)=(X^TX)^{-1}X^TY$ if the matrix $X^TX$ is invertible, and so an estimator of $r(x)$ at $x=(x_1,\ldots,x_p)\in\mathbb{R}^p$ is given by $\hat r_n(x)=x^T\hat\beta$.

Let $$\hat{\sigma}^2 = \frac{1}{\sigma^2}\sum_{i = 1}^n (Y_i - {\hat r}_n (x_i))^2.$$ I am stucking the problem: $\hat\sigma^2$ has $\chi^2_{n-p}$ distribution. How to prove the statement?

Answer 1

In the linear model with normal distributed errors you know that $$Y \sim \mathcal{N}_n(X\beta, \sigma^2I_n)$$ with $I_n$ is the identity matrix. Also you know that $$\sum_{i=1}^n\left(Y_i - \hat{r}(x_i)\right) = \|Y-X\hat{\beta}\|_2^2 = Y^TQ_LY$$ where $Q_L$ is the orthogonal projection into $L^\bot$ where $L := \{ X \beta : \forall\beta \in \mathbb{R}^p\}$. Note that $Q_L$ is a projection matrix.

($Q_L = I_n - P_L = I_n - X(X^TX)^{-1}X^T$)

Also it's important to take a closer look at the rank from $Q_L$. For that we know, that $P_L$ have rank $p$ if $X$ has full rank $p$. The $Q_L$ have rank $n-p$ because $I_n$ has rank $n$ and $P_L$ rank $p$ and $Q_L = I_n - P_L$ (for more details you ned linear algebra).

In the next step we take a look at $Z := \frac{1}{\sigma} Y$ which is $\mathcal{N}_n(X\beta/\sigma, I_n)$-distributed.

Now I use that for a $\mathcal{N}_n(\mu,I_n)$ - distributed random vector $X$ and a $n \times n$ - projection matrix $P$ with rank $n$ the expression $$X^TPX$$ is $\chi^2_n$ - distributed with noncentrality parameter $$\delta^2 = \mu^TP\mu$$

With that you can easy verify, that \begin{align*} \hat{\sigma}^2 &= \frac{1}{\sigma^2} Y^TQ_LY = (\frac{1}{\sigma}Y)^T Q_L (\frac{1}{\sigma}Y) = Z^TQ_LZ \end{align*} and with $\mathrm{rank}(Q_L) = n-p$ it follows that $$\hat{\sigma}^2 \sim \chi^2_{n-p}(\delta^2)$$ with noncentrality parameter $\delta^2 = \mu^TQ_L\mu$. At last we have to show that $\delta^2 = 0$.

In our case $\mu = X\beta / \sigma$. The vector $\beta \cdot \frac{1}{\sigma}$ is a vector in $\mathbb{R}^p$ and therefore $\mu \in L$. But when $\mu$ is in $L$ the product $Q_L \mu = 0$ because $\mu \bot L^\bot$ and with that $\delta^2 = 0$.

All in all $$\hat{\sigma}^2 \sim \chi^2_{n-p}$$

I hope you get it with that. To understand the proof in every detail you have to go very deep in probabilty theory and linear algebra (especially into projecitons).

Answer 2

We are given the linear model $Y=X\beta+\varepsilon$ where $X$ is an $n\times p$ constant matrix and the random vector $\epsilon$ consists of $n$ IID gaussians with mean zero and variance $\sigma^2$.

Introduce the hat matrix $H:=X(X^TX)^{-1}X^T$. Check that $H$ is symmetric and (1) $X\hat\beta = HY$ and (2) $HX=X$, so $$Y-X\hat\beta \stackrel{(1)}= Y-HY=(I-H)Y=(I-H)(X\beta +\varepsilon)\stackrel{(2)}=(I-H)\varepsilon, $$ and therefore the error sum of squares is $$\|Y-X\hat\beta\|^2=(Y-X\hat\beta)^T(Y-X\hat\beta)=[(I-H)\varepsilon]^T[(I-H)\varepsilon] = \varepsilon^T(I-H)\varepsilon, $$ the last equality following from the fact that $H^2=H$. Recalling that $\frac1\sigma\varepsilon$ is a vector of IID standard normal variables, we apply a theorem about quadratic forms involving standard normal variables and symmetric idempotent matrices to conclude that $\frac1{\sigma^2}\|Y-X\hat\beta\|^2$ has chi-squared distribution with degrees of freedom equal to the trace of $I-H$. Since the trace of the hat matrix equals the rank of $X$, $\operatorname{tr}(I-H)=n-\operatorname{tr}(H)=n-p$.