1
$\begingroup$

I am studying multiple linear regression and I am not able to understand a passage from my statistics textbook. The part that I do not understand can be formulated entirely in the language of linear algebra, as I will do here.

Let $X \in \mathbb{R}^{n \times m}$ an $n \times m$ matrix, with $n \geq m$ and with rank $m$. Let us denote $x_1, \ldots, x_m$ the $m$ column vectors. Let $P$ be the $n\times n$ matrix of the linear application describing the orthogonal projection from $\mathbb{R}^n$ onto the $m-1$-dimensional subspace \begin{equation*} \langle x_1, \ldots, x_{m-1}\rangle. \end{equation*} Let $I_n$ be the $n \times n$ identity matrix. Let us denote with $w^2_m$ the value at position $(m, m)$ of the $m \times m$ square matrix $(X^{\top}X)^{-1}$. Then \begin{equation*} w^2_m = \frac{1}{x_m^{\top}(I_n - P)x_m}. \end{equation*}

Why does the last equation hold?

Probably it can be useful to know that $P$ can be written as \begin{equation*} P = X_0(X_0^{\top}X_0)^{-1}X_0^{\top}, \end{equation*} where $X$ is the $n \times m-1$ matrix formed by the first $m-1$ columns of $X$.

$\endgroup$

1 Answer 1

1
$\begingroup$

Write $X$ as block matrix and calculate $X^{\top}X$: $$ X=\begin{pmatrix}X_0 & x_m\end{pmatrix}, $$ $$ X^{\top}X=\begin{pmatrix}X^\top_0 \\ x^\top_m\end{pmatrix}\begin{pmatrix}X_0 & x_m\end{pmatrix}=\begin{pmatrix}X^\top_0X_0 & X^\top_0x_m\\ x^\top_mX_0 & x^\top_m x_m\end{pmatrix}. $$ To find the value $w_m^2$ at position $(m,m)$ of $(X^{\top}X)^{-1}$, we need to divide determinant of $X^\top_0X_0$ by the determinant of $X^{\top}X$: $$ w_m^2=\frac{\det(X^\top_0X_0)}{\det(X^{\top}X)}. $$ With the help of determinant of block matrices https://en.wikipedia.org/wiki/Determinant#Block_matrices get $$ \det(X^{\top}X) = \textrm{det}(X^\top_0X_0)\cdot \textrm{det}\bigl(x_m^\top x_m-x_m^\top \underbrace{X_0 (X_0^\top X_0)^{-1}X_0^\top}_{P} x_m\bigr) $$ $$ =\det(X^\top_0X_0)\cdot \bigl(x_m^\top (I_n-P) x_m\bigr). $$ We skip last $\det$ since matrix $x_m^\top (I_n-P) x_m$ is $1\times 1$.

Finally, $$ w_m^2=\frac{\det(X^\top_0X_0)}{\textrm{det}(X^{\top}X)} = \frac{1}{x_m^\top (I_n-P) x_m}. $$

$\endgroup$
3
  • $\begingroup$ I do not understand why this is valid: "To find the value $w_m$ at position $(m,m)$ of $(X^{\top}X)^{-1}$, we need to divide determinant of $X^\top_0X_0$ by the determinant of $X^{\top}X$". Can you please develop it further? Thank you $\endgroup$
    – Nisba
    Dec 29, 2018 at 14:31
  • $\begingroup$ Look at en.wikipedia.org/wiki/… $\endgroup$
    – NCh
    Dec 29, 2018 at 14:54
  • $\begingroup$ Or look at en.wikipedia.org/wiki/… We need a cofactor to element $(m,m)$, and this is exactly the determinant of $X_0^\top X_0$. $\endgroup$
    – NCh
    Dec 29, 2018 at 15:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .