4
$\begingroup$

I want to ask how to show that Linear Regression is linear invariant? The problem is specified in the following picture: enter image description here

Here is the "solution" for the problem. But I really get confused by its second step: Why $(Z^TZ)^{-1} = Z^{-1}Z^{-T}$? The matrix Z is not a square matrix. I am wondering how this is legit? I will also greatly appreciate if anyone can give alternative solution to this problem. Thanks! enter image description here

$\endgroup$
2

1 Answer 1

5
$\begingroup$

You're right: distributing the inverse as they did only works if $Z$ is square, which it is usually not.

A valid solution is as follows: \begin{align} \theta' &= (Z^TZ)^{-1}Z^Ty \\ &= ((XA^T)^T(XA^T))^{-1}(XA^T)^Ty \\ & = (AX^TXA^T)^{-1}AX^Ty \\ & = [A^{-T}(X^TX)^{-1}A^{-1}]AX^Ty \\ &= A^{-T}(X^TX)^{-1}X^Ty \\ &= A^{-T}\theta \end{align} The trick here is not to break up the product $X^TX$ when distributing the inverse.

$\endgroup$
2
  • $\begingroup$ Thanks so much! Your solution works perfectly! I also wonder if it is possible to say $(Z^TZ)^{-1}=Z^{-1}Z^{-T}$ if we consider the inverse as the generalized inverse? I am thinking if the suggested solution is in fact also valid, but only that they used the definition of "inverse" differently than I thought $\endgroup$ Jun 9, 2018 at 1:18
  • $\begingroup$ Well with the usual pseudoinverse for this context, we actually have $Z^{\dagger} = (Z^TZ)^{-1}Z^T$ (when $Z$ has full column rank). However, I do not suspect that this is what they had in mind. $\endgroup$ Jun 9, 2018 at 1:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .