# Matrix rank and closed form solution to linear regression

We have learned that the closed form solution:

$$(X^T*X)^{-1}*X^T*\vec{y} = \vec{w}$$

for linear regression with X the $n*d$ design matrix, y the $n*1$ output and w the $d*1$ weight vector is only attainable, if $X^T*X$ is invertible. Invertibility means that $X^T*X$ has full rank, which in turn means that X has full rank. However only quadratic matrices can have full rank - does that mean that the closed form solution is only given for quadratic matrices?

No, not only square matrices can have full rank. A $n \times d$ matrix $X$ is said to have full rank if $rank(X) = \min \{n,d\}$
Consider $X = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$, this matrix has full rank, and $X^t X = [2]$ is invertible.
• I don't quite understand what you're asking now. If $n < d$ Then $X$ can hae at most rank $n$ and therefore $X^tX$ is a $d \times d$ matrix with rank $n < d$ and is therefore not a regular matrix. But this does not mean that there is no solution to the regression problem, it could very well be that the regression problem is just underdetermined and that you have a nontrivial solution space. Jul 28 '17 at 13:04
• Ah I see, if you're interested in a closed form that just picks one of the possible solutions for $n < d$ you should look into the singular value decomposition: If you want to solve $Xw=y$ as a least squares problem you can decompose $X=USV^t$ (the singular value decomposition) and use that to get $w = VS^{+}U^ty$. Jul 28 '17 at 18:47