# (Linear Regression) Proving that Linear Regression is linear invariant

I want to ask how to show that Linear Regression is linear invariant? The problem is specified in the following picture:

Here is the "solution" for the problem. But I really get confused by its second step: Why $(Z^TZ)^{-1} = Z^{-1}Z^{-T}$? The matrix Z is not a square matrix. I am wondering how this is legit? I will also greatly appreciate if anyone can give alternative solution to this problem. Thanks!

• Have a read of this: en.wikipedia.org/wiki/Inverse_element#Matrices Turns out $A^TA$ has this funny property. Jun 9, 2018 at 0:47
• amd: Can you explain why what you said holds for Z being a rectangular matrix? Jun 9, 2018 at 0:56

You're right: distributing the inverse as they did only works if $$Z$$ is square, which it is usually not.
A valid solution is as follows: \begin{align} \theta' &= (Z^TZ)^{-1}Z^Ty \\ &= ((XA^T)^T(XA^T))^{-1}(XA^T)^Ty \\ & = (AX^TXA^T)^{-1}AX^Ty \\ & = [A^{-T}(X^TX)^{-1}A^{-1}]AX^Ty \\ &= A^{-T}(X^TX)^{-1}X^Ty \\ &= A^{-T}\theta \end{align} The trick here is not to break up the product $$X^TX$$ when distributing the inverse.
• Thanks so much! Your solution works perfectly! I also wonder if it is possible to say $(Z^TZ)^{-1}=Z^{-1}Z^{-T}$ if we consider the inverse as the generalized inverse? I am thinking if the suggested solution is in fact also valid, but only that they used the definition of "inverse" differently than I thought Jun 9, 2018 at 1:18
• Well with the usual pseudoinverse for this context, we actually have $Z^{\dagger} = (Z^TZ)^{-1}Z^T$ (when $Z$ has full column rank). However, I do not suspect that this is what they had in mind. Jun 9, 2018 at 1:28