$I_m - AB$ is invertible if and only if $I_n - BA$ is invertible.

The problem is from Algebra by Artin, Chapter 1, Miscellaneous Problems, Exercise 8. I have been trying for a long time now to solve it but I am unsuccessful.

Let $A,B$ be $m \times n$ and $n \times m$ matrices. Prove that $I_m - AB$ is invertible if and only if $I_n - BA$ is invertible.

Please provide with only a hint.

-
Check out sylvester determinant theorem on Google. – Gautam Shenoy Nov 15 '12 at 6:29
Thank You. It makes the proof quite obvious now. :D – Jayesh Badwaik Nov 15 '12 at 6:32
If A, B are row/column vectors, then you can actually show this with induction. – Paxinum Nov 15 '12 at 7:03

Another method:

Let $C= {(I_m - AB)}^{-1}$. The matrix $BCA$ is $n\times n$.

$(I_n - BA)(BCA) = BCA - BABCA$

$= B(C - ABC)A$ $= B[(I_m - AB)C]A$ $= B(I_m)A\$ (by definition of $C$)

$= BA$

Hence,

$(I_n-BA)(BCA + I_n) = (I_n - BA) (BCA) + (I_n -BA) = BA + (I_n-BA) = I_n$

So, we get that the inverse of $I_n - BA$ is $I_n +BCA$.

-
 This is how I did the problem the first time, but then I forgot how I did it and was stuck. :P Thanks :-) – Jayesh Badwaik Nov 15 '12 at 7:41

There's no real need to actually invoke Sylvester's determinant theorem (although that certainly is much faster).

First show that the (non-zero) eigenvalues of $AB$ and the eigenvalues of $BA$ coincide. If you take the determinant of $I_m - AB$, then you have the characteristic polynomial of $AB$ evaluated at $\lambda = 1$. It follows that the determinant is zero if and only if $1$ is an eigenvalue of $AB$ if and only if $1$ is an eigenvalue of $BA$ if and only if $\det(I_n - BA) = 0$.

Note that a slight adaptation of this argument also provides a proof of Sylvester's determinant theorem different from the one given on Wikipedia.

-
 Thank you for another nice answer. – Jayesh Badwaik Nov 15 '12 at 6:53

Hint $\$ It's a consequence of Sylvester's determinant identity $\rm\:det(1 + AB) = det(1+BA),\:$ which has a very simple universal proof: over the polynomial ring $\rm\ \mathbb Z[A_{\,i\,j},B_{\,i\,j}\,]\$ take the determinant of $\rm\ (1+AB)\ A = A\ (1+BA)\ \$ then cancel $\rm\ det(A)\ \$ (which is valid since the ring is a domain). $\ \$ Extend to non-square matrices by padding appropriately with $0$'s and $1$'s to get square matrices. Note that the proof is purely algebraic - it does not require any topological notions (e.g. density).

Alternatively $\$ Proceed by way of Schur decomposition, namely

$$\rm\left[ \begin{array}{ccc} 1 & \rm A \\ \rm B & 1 \end{array} \right]\ =\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm B & 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm 0 & \rm 1-BA \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm A \\ \rm 0 & 1 \end{array} \right]$$

$$\rm\phantom{\left[ \begin{array}{ccc} 1 & \rm B \\ \rm A & 1 \end{array} \right]}\ =\ \left[ \begin{array}{ccc} 1 & \rm A \\ \rm 0 & 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1-AB & \rm 0 \\ \rm 0 & \rm 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm B & 1 \end{array} \right]$$

See my posts in this sci.math thread on 09 Nov 2007 for further discussion.

-