Try to realize that the state-space representation of a system is splitting your (physical) system up into 4 parts. In which the $A$ matrix represents the state of your system depending on a certain time $t$. $B$ represents the input-matrix which determines where on your system your input vector/scalar $u(t)$ will work. All you're checking with the controllability is to see if (independent of the $u(t)$ you choose) you will be able to control all the states of your system. This is to say, if there is a part of the physical system which you can't control, because you're "missing" an input somewhere.
Here are some definitions of controllability and reachability (which are closely related):
A state $x_1$ is called reachable at time $t_1$ if for some finite initial time $t_0$ there exists an input $u(t)$ that transfers the state $x(t)$ from the origin at $t_0$ to $x_1$.
A system is reachable at time $t_1$ if every state $x_1$ in the state-space is reachable at time $t_1$.
A state $x_0$ is controllable at time $t_0$ if for some finite time $t_1$ there exists an input $u(t)$ that transfers the state $x(t)$ from $x_0$ to the origin at time $t_1$.
A system is called controllable at time $t_0$ if every state $x_0$ in the state-space is controllable.
Try to think of it less in mathematical terms and more in a physical system.