$A$ and $B$ are $3\times 3$ real matrices such that $\operatorname{rank}(AB)=1$, then $\operatorname{rank}(BA$) can not be which of the following?

I was thinking about the problem that says:

If $A$ and $B$ are $3\times 3$ real matrices such that $\operatorname{rank}(AB)=1$, then $\operatorname{rank}(BA)$ can not be which of the following?
(a) $0$
(b) $1$
(c) $2$
(d) $3$.

My attempt: I have chosen suitable $3 \times 3$ matrices for $A$ and $B$ keeping in mind that $\operatorname{rank}(AB)=1$. Say for example if I take $A$ and $B$ to be $$A = \begin{pmatrix} 1 &2 &0 \\ 0 & 0 &0 \\ 0 & 0 &0 \end{pmatrix}$$ and $$B = \begin{pmatrix} -2 &1 &0 \\ 1 & 0 &0 \\ 0 & 0 &0 \end{pmatrix}$$
respectively, then I see $\operatorname{rank}(AB) = \operatorname{rank}(BA) = 1$. So, option (b) can not be correct. Do I have to keep choosing the matrices and then observe which of the option holds good. Is this kind of approach right to tackle the problem? I am looking for a direct way (e.g. application to some theorem) which can give me the result. I have also noticed that $AB$ and $BA$ are similar matrices as we see that $A^{-1}(AB)A=BA$. Is this observation going to help me in any way? Thanks in advance for your time.

-
How do you know det$A\neq0$? –  Sugata Adhya Dec 31 '12 at 15:55
Thanks for pointing out..Yes ,indeed det($A$) can be zero. –  learner Dec 31 '12 at 15:59

According to these properties of ranks listed on Wikiepdia : http://en.wikipedia.org/wiki/Rank_(linear_algebra)

If $A$ is $m \times n$

1. $\operatorname{rank}(AB) \leq \min(\operatorname{rank}(A), \operatorname{rank}(B))$ if $B$ is $n \times k$
2. $\operatorname{rank}(AB) = \operatorname{rank}(A)$ if $B$ is $n \times k$ of rank $n$

Note that in your situation $m$, $n$ and $k$ is $3$

By property (1) the only way to have $\operatorname{rank}(BA) = 3$ is to have both $\operatorname{rank}(A) = 3$ and $\operatorname{rank}(B) = 3$. Since $\min(\operatorname{rank}(A), \operatorname{rank}(B)) = \min(\operatorname{rank}(B), \operatorname{rank}(A))$.

But in that situation, we can apply property (2): $B$ is $3 \times 3$ of rank $3$ and so $\operatorname{rank}(AB) = \operatorname{rank}(A) = 3$.

So you can't have at the same time $\operatorname{rank}(BA) = 3$ and $\operatorname{rank}(AB) \neq 3$ using $3 \times 3$ matrix only.

-
thanks a lot.I have got it. –  learner Dec 31 '12 at 14:31

If you only want to solve the problem, then as the others have pointed out, (d) is impossible and hence the answer is (d). It is helpful, however, to constuct some examples to convince yourself that (a)-(c) are really possible (this is also a sanity check against misprints). For this purpose, you may consider the matrices $$A=\begin{pmatrix}1\\&1\\&&0\end{pmatrix},\ B=\begin{pmatrix}b&0&1\\0&0&0\\0&c&0\end{pmatrix}.$$ Then $\operatorname{rank}(AB)=1$. Now you may try to choose some appropriate values of $b$ and $c$ to make $\operatorname{rank}(BA)$ equal to $0,1$ or $2$.

-

The answer is (d), i.e. if $rank(AB)=1$, then $rank(BA)$ cannot be $3$. Let's prove it by contradiction. Suppose $A, B$ are $3\times 3$ matrices such that $rank(AB)=3$, then $AB$ is invertible. This implies that $A$ and $B$ are invertible (otherwise, if $A$ is not invertible, then $A$ could not be full rank, i.e. $rank(A)<3$. This would imply that $rank(AB)\leq rank(A)<3$, which contradicts $rank(AB)=3$. Similarly you can prove that $B$ is invertible). Then $BA$ is invertible since product of invertible matrices is invertible. This implies that $rank(BA)=3$, which contradicts to the assumption that $rank(AB)=1$.

To see that (a), (b), (c) are possible, we can construct some examples of $A$ and $B$. As you have shown that (b) is possible (In fact, we can just take $A$ to be any rank 1 matrix, i.e. $rank(A)=1$ and $B$ to be the identity. Then $AB=BA=A$ has rank $1$). I will leave other cases to you.

-
i have one question.If i want to prove it by contradiction,then should not i keep the condition that $rank(AB)=1$ intact and then take the supposition that if possible let us assume that $rank(BA)=3$ and then finally see what happens? with regards.. –  learner Dec 31 '12 at 14:26
@Paul: Is (a) possible? For rank$(BA)=0\implies BA=0\implies (AB)^2=0\implies$ rank$(AB)^2=0\implies$ min{rank($AB$), rank($AB$)} $=0\implies$rank $(AB)=0.$ –  Sugata Adhya Dec 31 '12 at 15:40

This question has lots of good answers.But I just want to share a very short answer.It may already be discussed in the previous answers,so really sorry to answer again.

$AB$ is a $3\times 3$ matrix,rank$(AB)=1\ne3\Rightarrow AB$ is not of full rank$\Rightarrow \operatorname{det}(AB)=0\Rightarrow \operatorname{det}(BA)=0\Rightarrow BA$ is not of full rank$\Rightarrow$rank$(BA)\ne3$

-