Consider the linear regression model:

$$Y_i=r(x_i)+\varepsilon_i\equiv\sum_{j = 1}^p x_{ij} \beta _j + \varepsilon _i,\quad i=1,\ldots,n.$$

where $x_1,\ldots,x_n\in \mathbb{R}^p$ are fixed, $E(\varepsilon_i)=0$, $\operatorname{Var}(\varepsilon_i)=\sigma^2$. Denote $Y=(Y_1,\ldots,Y_n)^T$, $\beta=(\beta_1,\ldots,\beta_p)^T$, $X=(x_{ij})_{n\times p}$. As is known, $\hat \beta = \arg\min\limits_{\beta \in \mathbb{R}^p} (Y - X\beta)^T (Y - X\beta)=(X^TX)^{-1}X^TY$ if the matrix $X^TX$ is invertible, and so an estimator of $r(x)$ at $x=(x_1,\ldots,x_p)\in\mathbb{R}^p$ is given by $\hat r_n(x)=x^T\hat\beta$.

Let $$\hat{\sigma}^2 = \frac{1}{\sigma^2}\sum_{i = 1}^n (Y_i - {\hat r}_n (x_i))^2.$$ I am stucking the problem: $\hat\sigma^2$ has $\chi^2_{n-p}$ distribution. How to prove the statement?


In the linear model with normal distributed errors you know that $$Y \sim \mathcal{N}_n(X\beta, \sigma^2I_n)$$ with $I_n$ is the identity matrix. Also you know that $$\sum_{i=1}^n\left(Y_i - \hat{r}(x_i)\right) = \|Y-X\hat{\beta}\|_2^2 = Y^TQ_LY$$ where $Q_L$ is the orthogonal projection into $L^\bot$ where $L := \{ X \beta : \forall\beta \in \mathbb{R}^p\}$. Note that $Q_L$ is a projection matrix.

($Q_L = I_n - P_L = I_n - X(X^TX)^{-1}X^T$)

Also it's important to take a closer look at the rank from $Q_L$. For that we know, that $P_L$ have rank $p$ if $X$ has full rank $p$. The $Q_L$ have rank $n-p$ because $I_n$ has rank $n$ and $P_L$ rank $p$ and $Q_L = I_n - P_L$ (for more details you ned linear algebra).

In the next step we take a look at $Z := \frac{1}{\sigma} Y$ which is $\mathcal{N}_n(X\beta/\sigma, I_n)$-distributed.

Now I use that for a $\mathcal{N}_n(\mu,I_n)$ - distributed random vector $X$ and a $n \times n$ - projection matrix $P$ with rank $n$ the expression $$X^TPX$$ is $\chi^2_n$ - distributed with noncentrality parameter $$\delta^2 = \mu^TP\mu$$

With that you can easy verify, that \begin{align*} \hat{\sigma}^2 &= \frac{1}{\sigma^2} Y^TQ_LY = (\frac{1}{\sigma}Y)^T Q_L (\frac{1}{\sigma}Y) = Z^TQ_LZ \end{align*} and with $\mathrm{rank}(Q_L) = n-p$ it follows that $$\hat{\sigma}^2 \sim \chi^2_{n-p}(\delta^2)$$ with noncentrality parameter $\delta^2 = \mu^TQ_L\mu$. At last we have to show that $\delta^2 = 0$.

In our case $\mu = X\beta / \sigma$. The vector $\beta \cdot \frac{1}{\sigma}$ is a vector in $\mathbb{R}^p$ and therefore $\mu \in L$. But when $\mu$ is in $L$ the product $Q_L \mu = 0$ because $\mu \bot L^\bot$ and with that $\delta^2 = 0$.

All in all $$\hat{\sigma}^2 \sim \chi^2_{n-p}$$

I hope you get it with that. To understand the proof in every detail you have to go very deep in probabilty theory and linear algebra (especially into projecitons).


We are given the linear model $Y=X\beta+\varepsilon$ where $X$ is an $n\times p$ constant matrix and the random vector $\epsilon$ consists of $n$ IID gaussians with mean zero and variance $\sigma^2$.

Introduce the hat matrix $H:=X(X^TX)^{-1}X^T$. Check that $H$ is symmetric and (1) $X\hat\beta = HY$ and (2) $HX=X$, so $$Y-X\hat\beta \stackrel{(1)}= Y-HY=(I-H)Y=(I-H)(X\beta +\varepsilon)\stackrel{(2)}=(I-H)\varepsilon, $$ and therefore the error sum of squares is $$\|Y-X\hat\beta\|^2=(Y-X\hat\beta)^T(Y-X\hat\beta)=[(I-H)\varepsilon]^T[(I-H)\varepsilon] = \varepsilon^T(I-H)\varepsilon, $$ the last equality following from the fact that $H^2=H$. Recalling that $\frac1\sigma\varepsilon$ is a vector of IID standard normal variables, we apply a theorem about quadratic forms involving standard normal variables and symmetric idempotent matrices to conclude that $\frac1{\sigma^2}\|Y-X\hat\beta\|^2$ has chi-squared distribution with degrees of freedom equal to the trace of $I-H$. Since the trace of the hat matrix equals the rank of $X$, $\operatorname{tr}(I-H)=n-\operatorname{tr}(H)=n-p$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.