If $AB\not=BA,$ then do $A$ and $B$ have to be matrices Is this solvable? Or are there other things that fit the bill for $A$ and $B$?
 A: In the study of abstract algebra, we do not limit ourselves to numbers. You have already noted that matrices act different than numbers, but mathematicians study general algebraic objects called groups. Invertible matrices and real numbers are groups, and @IvoTerek has noted another group called the quaternion group. But there are lots of others. A group is a set $G$ with an operation $\times$ so that if $a,b\in G$, $a\times b\in G$. There are a few rules for a group:


*

*The group operation is transitive: $a\times (b\times c)=(a\times b)\times c.$

*The group has an identity element $e$ so that $ae=ea=a$ for all $a\in G$. 

*Every $a\in G$ has an inverse $a^{-1}$ so that $a^{-1}a=aa^{-1}=e$.


Note that none of the group rules require that $ab=ba$, so lots of groups fit the description you give. In fact, groups that do have the property are rather rare (they are called Abelian groups). Several examples of non-Abelian groups you might look up are:


*

*Dihedral groups

*Permutation groups

*Quaternion groups.

A: $AB\ne BA$ is possible if $A,B$ are elements of a noncommutative ring, and there are noncommutative rings that cannot be represented as ring of matrices. As an exemple you can see the Weyl algebra.
Note that  every simple ring that is finite-dimensional over a division ring (as the quaternions) can be represented as a matrix ring ( see Artin-Wedderburn theorem)
A: Matrices are best thought of as linear transformations. The matrix product $XY$ is the linear transformation obtained by first doing $Y$ and then doing $X$. The matrix product $YX$ is the linear transformation obtained by first doing $X$ and then doing $Y$. The order matters. When the order matters, we say that the operations are non-commutative.
There are many examples of non-commutative operations, i.e. doing $A$ and then doing $B$ gives a different result to first doing $B$ and then doing $A$. For example, $A$ and $B$ could be permutations. Let $A = (123)$, i.e. $1 \to 2$, $2\to 3$ and $3\to 1$. Let $B=(13)$, i.e. $1 \to 3$ and $ 3\to 1$.
\begin{eqnarray*}
AB &=& (123)(13) &=& (23) \\
BA &=& (13)(123) &=& (12)
\end{eqnarray*}
