Find a Jordan Canonical Form Let $A$ and $B$ be matrices over the real numbers, such that $A$ is $3 \times 5$ and $B$ is $5 \times 3$, and the product $AB$ is
$$
\left( \begin{array} \\ 1& 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right).
$$
I want to find the Jordan canonical form of $BA$.
I am rather stuck on this. The only information that I know to go on is that $BA$ is $5 \times 5$ and it's rank is at most 3, so $0$ is an eigenvalue of it and it's multiplicity is at least $2$.
 A: If $ABv=\lambda v$, then $BA(Bv)=B(ABv)=B(\lambda v)=\lambda(Bv)$, so if $v$ is an eigenvector of $AB$ with eigenvalue $\lambda$, then $Bv$ is an eigenvector of $BA$ with eigenvalue $\lambda$.  If $\lambda\neq 0$, then $Bv\neq 0$ because $A(Bv)\neq 0$, so this shows that, with the exception of $0$, $AB$ and $BA$ have the same eigenvalues with the same geometric multiplicities.
Similarly, one can show that the characteristic polynomials of $AB$ and $BA$ are the same up to an additional factor of $t^k$, so the algebraic multiplicities of the eigenvalues are the same, except for $0$.
In low dimensions, this information is enough to determine the Jordan Normal Form, except for the $0$ block.
For that, we can look at the minimal polynomial.  We have that $(AB-I)^2=0$, and so $B(AB-I)^2A=(BA-I)^2BA=0$.  Because this has a single factor of $BA$, we cannot have a block of $0$ of size bigger than $1$.
This argument actually shows that $AB$ and $BA$ have the same sized largest block for each eigenvalue except for $0$, and the maximum block size for $0$ can differ by at most $1$.
To see that the maximal block size for $0$ can change, consider $A=(0,1)$, $B=(1,0)^T$.

As an alternative (and somewhat more useful way, as it will give results in higher dimensions) to looking at characteristic polynomials.  We have by induction that $B(AB-\lambda I)^k=(BA-\lambda I)^k B$, and so if $v$ is an approximate eigenvector of $AB$ with eigenvalue $\lambda$ and order $k$, then $Bv$ is an approximate eigenvector of $BA$ with eigenvalue $\lambda$ and order $k$. I think this can be used to show that when $\lambda\neq 0$, the approximate eigenspaces have the same dimensions, and so all the block sizes will be the same for the non-zero eigenvalues.  However, I have not checked the details, so this may not go through.
A: As you noted, we have
$$r(BA) \le \max\{r(A),r(B)\} \le 3$$
so $0$ is an eigenvalue of $BA$ with algebraic multiplicity at least $2$.
Clearly $\sigma(AB) = \{1\}$ and $\dim\ker (AB-I) = 2$. If $(AB-I)x = 0$ then
$$(BA-I)Bx = B(AB-I)x = 0$$
which implies that $B(\ker(AB-I)) \subseteq \ker (BA-I)$. From
$$\ker B\cap \ker (AB-I) = \{0\}$$
we get that $B|_{\ker(BA-I)}$ is injective. Therefore $$\dim\ker (BA-I) \ge \dim B(\ker(AB-I)) = \dim\ker(AB-I)=2.$$
Similarly we get $A(\ker(BA-I))\subseteq \ker(AB-I)$ and $A|_{\ker(AB-I)}$ is injective so $$2 = \dim \ker(AB-I) \ge \dim A(\ker(BA-I)) = \dim \ker(BA-I)$$
from which we conclude $\dim \ker(BA-I)=2$.
Since $0$ and $1$ are eigenvalues of $BA$ with geometric multiplicities $\ge 2$, the spectrum of $BA$ with algebraic multiplicities is $$\sigma(BA) = \{0,0,1,1,?\}.$$
To determine the last eigenvalue note that $$2\,+\,?=\operatorname{Tr}(BA) = \operatorname{Tr}(AB)=3$$
so the last eigenvalue is also $1$. Hence it must be $\dim\ker(BA) = 2$ so there are precisely two $0$-blocks of size $1$. From $\dim\ker(BA-I) = 2$ we get that there are two $1$-blocks and that $BA$ is not diagonalizable so there is one $1$-block of size one and one $1$-block of size $2$:
$$\begin{bmatrix} 1 & 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0\end{bmatrix}.$$
A: Let $O_1, O_2, O_3$ are $2\times5, 5\times2, 2\times 2$ zero matrices, respectively.
Define
$$ A_1=\binom{A}{0_1}, B_1=(B, 0_2) $$
be $5\times 5$ matrices. Then
$$ B_1A_1=BA, A_1B_1=\bigg(\begin{matrix}AB&0_2\\0_1&0_3\end{matrix}\bigg).$$
By
$$ \det(B_1A_1-\lambda I_5)=\det(A_1B_1-\lambda I_5) $$
one has
$$ \det(BA-\lambda I_5)=\det\bigg[\bigg(\begin{matrix}AB&0_2\\0_1&0_3\end{matrix}\bigg)-\lambda I_5\bigg]=\lambda^2\det(AB-\lambda I_3)=\lambda^2(\lambda-1)^3. $$
This implies that $BA$ has eigenvalues $0$ and $1$.
Clearly $BA$ has eigenvectors $v_4=(0,0,0,1,0)^T,v_5=(0,0,0,0,1)^T$ of eigenvalue $0$ since
$$ (BA)v_4=B_1A_1v_4=0, (BA)v_5=B_1A_1v_5=0. $$
Let $w_1=(1,0,0)^T,w_2=(0,1,0)^T,w_1=(0,0,0)^T$ and then
$$ ABw_1=v_1,ABw_2=w_1,ABw_3=w_3, $$
namely $AB$ has eigenvectors $w_1,w_3$ are of eigenvalue 1 and $w_2$ as a generalized eigenvector of 1.
Define
$$ v_1=Bw_1,v_2=Bw_2,v_3=Bw_3. $$
Then
$$ (BA)v_1=B(ABw_1)=Bw_1=v_1, (BA)v_2=B(ABw_2)=Bw_2=v_1, (BA)v_3=B(ABw_3)=Bw_3=v_3 $$
namely $BA$ has eigenvectors $v_1,v_3$ are of eigenvalue 1 and $v_2$ as a generalized eigenvector of 1. Let
$$ S=(v_1,v_2,v_3,v_4,v_5) $$
and then
$$ S^{-1}(BA)S=\left(\begin{matrix}
1&1&0&0&0\\
0&1&0&0&0\\
0&0&1&0&0\\
0&0&0&0&0\\
0&0&0&0&0\end{matrix}\right). $$
