2
$\begingroup$

We have learned that the closed form solution:

$$(X^T*X)^{-1}*X^T*\vec{y} = \vec{w}$$

for linear regression with X the $n*d$ design matrix, y the $n*1$ output and w the $d*1$ weight vector is only attainable, if $X^T*X$ is invertible. Invertibility means that $X^T*X$ has full rank, which in turn means that X has full rank. However only quadratic matrices can have full rank - does that mean that the closed form solution is only given for quadratic matrices?

$\endgroup$
1
$\begingroup$

No, not only square matrices can have full rank. A $n \times d$ matrix $X$ is said to have full rank if $rank(X) = \min \{n,d\}$

Consider $X = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$, this matrix has full rank, and $X^t X = [2]$ is invertible.

$\endgroup$
4
  • $\begingroup$ Then for n x d matrix X, if the matrix X has full rank and n < d, the matrix is left-invertible, and if n > d, the matrix is right-invertible? How comes that in case of linear regression, if n < d, the above closed form solution does not hold? I thought this was due to the lack deficiency $\endgroup$
    – user66280
    Jul 28 '17 at 10:55
  • 1
    $\begingroup$ I don't quite understand what you're asking now. If $n < d$ Then $X$ can hae at most rank $n$ and therefore $X^tX$ is a $d \times d$ matrix with rank $n < d$ and is therefore not a regular matrix. But this does not mean that there is no solution to the regression problem, it could very well be that the regression problem is just underdetermined and that you have a nontrivial solution space. $\endgroup$
    – flawr
    Jul 28 '17 at 13:04
  • $\begingroup$ I understand this, but my question was if the closed form solution is available if n < d. As I understand your answer, it is not, since the closed form solution requires X^tX to be regular (i.e. invertible). Thank you! $\endgroup$
    – user66280
    Jul 28 '17 at 13:28
  • 1
    $\begingroup$ Ah I see, if you're interested in a closed form that just picks one of the possible solutions for $n < d$ you should look into the singular value decomposition: If you want to solve $Xw=y$ as a least squares problem you can decompose $X=USV^t$ (the singular value decomposition) and use that to get $w = VS^{+}U^ty$. $\endgroup$
    – flawr
    Jul 28 '17 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy