Determine all matrices $M\in M_3(\mathbb K)$ such that $M^2 = O_3$ 
How to determine all matrices $M\in M_3(\mathbb K)$ such that $M^2=O_3$?

I tried saying that if 
$$M=\begin{pmatrix}
a & b & c\\
d& e & h\\
i & j & k\\
\end{pmatrix}$$
then$$M^2=\begin{pmatrix}
a & b & c\\
d& e & h\\
i & j & k\\
\end{pmatrix}\begin{pmatrix}
a & b & c\\
d& e & h\\
i & j & k\\
\end{pmatrix}$$
therefore:
\begin{cases}
a^2 +bd + gc =0\\
ab+be + ch =0\\
ac + bf + ci=0\\
...
\end{cases}
but it seems to be the wrong way.
The answer given which I don't understand was:
Let suppose that $M\neq O_3$ and let be $\phi$ the associated endomorphism canonically associated to $M$. 
Then it exists a vector $\vec x$ such that $\phi (x) =0$. 
We have $Im\phi\subset \ker \phi$ (but I don't understand why). 
and because $\dim Im \phi+\dim\ker\phi=3$ then necessarily $\dim Im \phi=1$ and $\dim\ker\phi=3$ (Which I think is wrong). 
Because $\phi(x)\in\ker\phi$ (I still don't understand why), $z\in\ker\phi$ exists such that {\phi (x),x} be a basis of $\ker\phi$.
The family $F=\{x,\phi x,z\}$ is free and the matrix $\phi$ regarding this basis is: 
\begin{pmatrix}
0 & 0 & 0\\
1 & 0 & 0\\
0 & 0 & 0
\end{pmatrix} 
 A: When you say that $M^2 = 0$, you're saying that 
$$
M(Mx) = 0
$$
for every $x \in  \Bbb{R}^3$. 
Since the set of all $Mx$ (as $x$ ranges over $ \Bbb{R}^3$) is exactly the image of $\phi$, and $M$ times each of these is zero, each of them (i.e., each item in the image of $\phi$) must be in the kernel of $\phi$. 
The thing you think is wrong is, in fact, wrong: they should have written dim ker $\phi \ge 1$ and dim im $\phi \le 2$. 
Let's walk through that proof step by step, with a few extra details. 

Let suppose that $M\neq O_3$ and let be $\phi$ the associated
  endomorphism canonically associated to $M$.

There are two possibilities: $M = 0$ and $M \ne 0$. The first one is easy, so let's look at the second, and let's define $\phi:  \Bbb{R}^3 \to \Bbb{R}^3 : x \mapsto Mx$. 

Then there exists a vector $\vec x$ such that $\phi (x) =0$. 

Because $M^2 = 0$, we know that $M(Mx) = 0$ for every $x$. Pick a nonzero $x$. Then either (1) $Mx = 0$, or (2) $y = Mx$ is a nonzero vector. But then since $M(Mx) = My = 0$, $y$ is a nonzero vector in the kernel of $\phi$. So in both cases, $\phi$ has a nonzero element in its kernel. So dim ker $\phi \ge 1$. 

We have $Im\phi\subset \ker \phi$ (but I don't understand why). 

Because $Im \phi$ consists of all vectors $Mx$ where $x \in R^3$, and for all vectors $x$, we have $M(Mx) = 0,$ we see that for any vector $y = Mx \in Im \phi$, we have $\phi(y) = My = MMx = 0$, so that $y \in Ker \phi$. 

and because $\dim Im \phi+\dim\ker\phi=3$ then necessarily $\dim Im \phi=1$ and $\dim\ker\phi=3$ (Which I think is wrong). 

This is wrong. We only have $\dim Im \phi \ge 1$ and $\dim Ker \phi \le 2$. But suppose that $\dim Im \phi = 2$. Then $\dim Ker \phi = 1$. But since $Im \phi \subset ker \phi$, this is impossible (by dimension). So we must have $\dim Im \phi = 1$ and $\dim ker \phi = 2$. 

Because $\phi(x)\in\ker\phi$ (I still don't understand why), $z\in\ker\phi$ exists such that {\phi (x),x} be a basis of $\ker\phi$.

Frankly, the statement above makes no sense to me: the conclusion seems to be unrelated to $z$. But the idea is, I think, this:


*

*If you start with a nonzero vector $x$ with $\phi(x) \ne 0$, then there's some other vector $z \in Ker \phi$ with the property that $\phi(x), z$ is a basis of the kernel. Why? Because the kernel is two-dimensional, and any linearly independent set (like the singleton $\{ \phi(x) \}$) can be extended to a basis. 

*The triple $x, \phi(x), z$ is a basis of $R^3$ (this takes some proving)

*The matrix of $\phi$ with respect to this basis is the one given in the next line. 

The family $F=\{x,\phi x,z\}$ is free and the matrix $\phi$ regarding
  this basis is: 
\begin{pmatrix} 0 & 0 & 0\\ 1 & 0 & 0\\ 0 & 0 & 0 \end{pmatrix}

So the only question that remains is "can you prove that those three vectors form a basis of 3-space?" 
Addendum after comments
We have $x, \phi(x), z$, where $x$ and $\phi(x)$ are nonzero, and $z$ is an element of $ker \phi$ that's independent of $\phi(x)$. We'd like to show that these three form a basis of $\mathbb R^3$. Since the dimension is three, it suffices to show that they're linearly independent. So suppose that
$$
\alpha x + \beta \phi(x) + \gamma z = 0.
$$
We must show that the coefficients (in greek letters) are all zero. Apply the transformation $\phi$ (which satisfies $\phi(\phi u) = 0$ for all $u$) to the statement above to get
\begin{align}
\phi(\alpha x + \beta \phi(x) + \gamma z) &  = \phi(0) \\
\phi(\alpha x) + \phi(\beta \phi(x)) + \phi(\gamma z)) &  = 0 \\
\alpha \phi(x) + \beta \phi(\phi(x)) + \gamma \phi(z)) &  = 0 \\
\alpha \phi(x) + \beta 0 + \gamma 0 &  = 0 \\
\end{align}
where the first three steps follow from linearity of $\phi$, and the last because $\phi^2 = 0$ and because $z \in ker ~\phi$. From this, we conclude $\alpha = 0$. 
Now we have $\beta \phi(x) + \gamma z = 0$. But $\phi(x)$ and $z$ were chosen to be a basis of $ker~ \phi$, hence are linearly independent. So $\beta = \gamma = 0$ and we're done. The three vectors really do form a basis of $\mathbb R^3$. 
What about the matrix of $\phi$ in this basis? Let's look at $\phi(u)$ for $u = x$ and write it in terms of the basis $x, \phi(x), z$, and then do the same for $u = \phi(x)$ and for $u = z$. Well: 
$$
\phi(x) = 0 x + 1 \phi(x) + 0 z
$$
so the first column of the matrix contains the coefficients $0,1,0$, and is 
$$
\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}.
$$
What about $u = \phi(x)$. Well, in this case, $\phi(u) = \phi(\phi(x)) = 0$, so we have
$$
\phi(\phi(x)) = 0 x + 0 \phi(x) + 0z.
$$
So the second column is 
$$
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}.
$$
And finally, for $u = z$, we know that $z \in ker~\phi$, so $\phi z = 0$. Thus the third column is also 
$$
\begin{bmatrix}
0\\
0\\
0
\end{bmatrix}.
$$
and the matrix (in the $x, \phi(x), z$ basis) overall is
$$
\begin{bmatrix}
0 & 0 & 0\\
1& 0 & 0\\
0& 0 & 0
\end{bmatrix}.
$$
A: $M=O_{3}$ is obviously such a matrix. Let suppose $M\neq O_{3}$ and let $\phi$ be the morphism canonically associated to $M$. 
As $M^{2}=O_{3}$, it means that whatever $x\in\mathbb{K}^{3}$, we have $\phi(\phi(x))=0$, hence $\phi(x)\in\text{ Ker }\phi$ for any $x\in\mathbb{K^{3}}$, which explicitly means that $\text{ Im }\phi\subset\text{ Ker }\phi$ since $\text{ Im }\phi :=\{\phi(x)\mid x\in\mathbb{K^{3}}\}$.
As $M\neq O_{3}$, there exists at least one $x\in\mathbb{K}^{3}$ such that $\phi(x)\neq 0$, i.e. $\text{ Im }\phi\neq\{0\}$, so that $\text{ dim }\text{ Im }\phi$ is at least $1$. Since we also have 
$$\text{ dim }\text{ Im }\phi + \text{ dim }\text{ Ker }\phi = \text{ dim }\mathbb{K^{3}}=3$$
and because $1\le \text{ dim }\text{ Im }\phi\le \text{ dim }\text{ Ker }\phi$, the only possibility is $\text{ dim }\text{ Im }\phi=1$ and $\text{ dim }\text{ Ker }\phi=2$.
A: Your method is fine, and it has its advantages, but it's perhaps computationally unpleasant.
Since $0 = \phi^2 = \phi \circ \phi$, for all $x \in \Bbb K^3$ we have $\phi(\phi(x)) = 0$, so $\phi(\operatorname{im} \phi) = 0$, or equivalently, $$\operatorname{im} \phi \subseteq \ker \phi$$ as claimed, and in particular we have
$$\dim \operatorname{im} \phi \leq \dim \ker \phi.$$ By the Rank-Nullity Theorem we have
$$\dim \operatorname{im} \phi + \dim \ker \phi = 3 ,$$ so $\dim \operatorname{im} \phi \leq 1$. On the other hand, since $\phi \neq 0$, we must have $\dim \operatorname{im} \phi > 0$, so $\dim \operatorname{im} \phi = 1 .$

Here's an alternative method for addressing the problem, which in particular uses the Jordan Canonical Form: Notice that if $M^2 = 0$ and $P$ is invertible, then $(P M P^{-1})^2 = 0$, that is the property is invariant under similarity. On the other hand, $M^2 = 0$ implies that all of the eigenvalues of $M$ are zero, and the only Jordan blocks that square to $0$ are $0$ and $\pmatrix{0&1\\0&0}$. So, up to similarity the only matrices with square the zero matrix are the zero matrix itself and $$J := \pmatrix{0&1&0\\0&0&0\\0&0&0} .$$ (It's not hard to build a permutation matrix that conjugates $J$ to the matrix of the form in the question.)
So, all nonzero $3 \times 3$ matrices with square zero can be written as
$$P^{-1} J P$$ for some invertible $P$.
