1
$\begingroup$

It is well-known that for the general linear system

$$\mathbf{X}\mathbf{a}=\mathbf{y}$$

where $\mathbf{X}$ is a rectangular matrix and $\mathbf{a}$ and $\mathbf{y}$ are column vectors of compatible sizes, the solution given by the Moore-Penrose pseudoinverse $\mathbf{a}=\mathbf{X}^+\mathbf{y}$ always exist and

  1. if $\mathbf{y}$ is in the column space of $\mathbf{X}$ (i.e. the system is satisfiable), $\mathbf{a}=\mathbf{X}^+\mathbf{y}$ is the solution with minimum norm.

  2. if $\mathbf{y}$ is not in the column space of $\mathbf{X}$ (i.e. the equality is not satisfiable), then $\mathbf{a}=\mathbf{X}^+\mathbf{y}$ is the best approximation for the equality, i.e. $\mathbf{a}=\mathbf{X}^+\mathbf{y} = \text{argmin}_\mathbf{a} \|\mathbf{Xa}-\mathbf{y}\|^2$

furthermore if $\mathbf{X}$ has linearly-independent columns then $\mathbf{a}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$


Now regarding the Matrix Equation

$$\mathbf{XA}=\mathbf{Y}$$ where $\mathbf{A}$ is the unknown Matrix and $\mathbf{X}$ and $\mathbf{Y}$ are matrices of compatible sizes

Can we also say that $\mathbf{A}=\mathbf{X}^+\mathbf{Y}$ has the same properties as before? Namely

  1. if each column of $\mathbf{Y}$ belongs to the column space of $\mathbf{X}$ then $\mathbf{A}=\mathbf{X}^+\mathbf{Y}$ is the solution with minimum (matrix) norm?

  2. if some column of $\mathbf{Y}$ does not belong to the column space of $\mathbf{X}$ then $\mathbf{A}=\mathbf{X}^+\mathbf{Y}$ is the best approximation for the system, i.e. $\mathbf{A}=\mathbf{X}^+\mathbf{Y}=\text{argmin}_\mathbf{A}\|\mathbf{XA}-\mathbf{Y}\|^2$ for some matrix norm

Furthermore if $\mathbf{X}$ has linearly-independent columns then $\mathbf{A}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y}$?

$\endgroup$
  • 1
    $\begingroup$ You can write the matrix equation as a system of vector-equations as above. So, you have similar results. Not sure however, if the norm-property holds for this. $\endgroup$ – Peter May 2 '18 at 13:12
  • 1
    $\begingroup$ +1. I think all of that is true, basically for the reason @Peter said: multivariate regression is just a bunch of univariate regressions stacked together. The norm in (1) and (2) should be Frobenius norm (sum of squared elements), and the squared Frobenius norm is the sum of squared L2 norms across columns. I did not write it down carefully now, but I am pretty sure everything you said will hold. $\endgroup$ – amoeba May 8 '18 at 11:44
  • $\begingroup$ You have full derivation in math.stackexchange.com/a/2714335/33. $\endgroup$ – Royi May 25 '18 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.