Example of two matrices $A,B$ I need to find two $2\times 2$ matrices with a real values such that $$A^2=B^2=(AB)^4=I\neq (AB)^2.$$
Also in other problem I have to find such matrices $A,B$ that $$A^2 = B^2 = I$$ and $(AB)^n\neq I$ for all $n\in \mathbb{N}_+$. To be honest I don't see any way to solve this other than brutally multiplying arbitrary matrix $2\times 2$. I'm sure there's a smart way to do this. I would be grateful for any tip for either of these problems.
 A: A possibility it to go for matrices like $M(t)=\begin{pmatrix}\cos(t)&\sin(t)\\\sin(t)&-\cos(t)\end{pmatrix}$ for which $M(t)^2=I$
Take $A=M(a)$ and $B=M(b)$, you will see that $ \begin{cases}(AB)^2=R(2(a-b))\\(AB)^4=R(4(a-b))\end{cases}$
where $R(t)=\begin{pmatrix}\cos(t)&-\sin(t)\\\sin(t)&\cos(t)\end{pmatrix}$ is the usual rotation matrix of angle $t$.
The idea is thus to take $4(a-b)=2\pi$ or $a=b+\frac{\pi}{2}$ and it works, you'll get $(AB)^2=-I$ and $(AB)^4=I$
In fact with $J=\begin{pmatrix}1&0\\0&-1\end{pmatrix}$ then $M=RJ$ but $J$ and $R$ do not commute.
A: For your first question, let $C=(AB)^2$. Then $A^2=B^2=C^2=I$. Hence all of $A,B$ and $C$ are diagonalisable and their possible spectra are $\{1,1\},\{1,-1\}$ and $\{-1,-1\}$.
As $B^2\neq (AB)^2$, $A$ cannot be equal to $\pm I$. Hence the spectrum of $A$ can only be $\{1,-1\}$ and we may assume that $A=\pmatrix{1&0\\ 0&-1}$.
Also, as $C\ne I$ is a diagonalisable matrix with $\det(C)=\det(AB)^2\ge0$, its spectrum must be $\{-1,-1\}$. Hence $C=-I$.
The problems now reduces to solving $B^2=I$ and $(AB)^2=C=-I$ simultaneously. As the first equation implies that $B=B^{-1}$, the second one can be rewritten as
$$
B\pmatrix{1&0\\ 0&-1}B^{-1}=\pmatrix{-1&0\\ 0&1}.
$$
Obviously these two equations are satisfied by the permutation matrix $B=\pmatrix{0&1\\ 1&0}$. We are now done.

For your second question, a similar argument to the above answer shows that the spectra of both $A$ and $B$ must be $\{1,-1\}$. Now the inequation $(AB)^n\ne I$ is satisfied for all integer $n\ge1$ if no eigenvalues of $AB$ lie on the unit circle. So, you may pick a random nonsingular matrix $P$ and try
$$
A=\pmatrix{1&0\\ 0&-1},\ B=P\pmatrix{1&0\\ 0&-1}P^{-1}.
$$
For instance,
$$
P=\pmatrix{2&1\\ 1&1}\Rightarrow AB=\pmatrix{3&-4\\ -2&3}.
$$
The eigenvalues of $AB$ are then $3\pm\sqrt{8}$, which don't lie on the unit circle. Hence $(AB)^n\ne I$ for any nonzero integer $n$.
A: We can write $A=UDU^{-1}$ where $D$ is a diagonal matrix with the eigenvalues on its diagonal. Note that those eigenvalues can be complex numbers.
If follows from $A^2=I$ that $D^2=I$. Therefore $A$ is constrained to the eigenvalues $\pm 1$. If both eigenvalues are $+1$, then $A=I$. And if both eigenvalues are $-1$, then $A=-I$. In both cases we would have  $(AB)^2=B^2=I$, which is a contradiction. Therefore $A$ is of the form $U\begin{pmatrix}1&0\\0&-1\end{pmatrix}U^{-1}$. The same argument applies to $B$, which must therefore be of the same form.
In the same fashion we can write $AB=W\Lambda W^{-1}$ where $\Lambda$ is again a diagonal matrix. If follows from $(AB)^4=I$ that $\Lambda^4=I$, and therefore the eigenvalues of $AB$ are in $\{\pm 1,\pm i\}$. Since $(AB)^2\ne I$, at least one of those eigenvalues must be $\pm i$. Since a real matrix with a complex eigenvalue must also have the conjugate eigenvalue, $AB$ must have eigenvalues $+i$ and $-i$.
A possible choice is to pick $AB$ as the rotation matrix over 90 degrees, and to pick $A$ and $B$ as reflection matrices such that their axes of reflection make an angle of 45 degrees, so that their product is the desired rotation matrix. That is, we can pick $A=\begin{pmatrix}1&0\\0&-1\end{pmatrix}$ and $B=\begin{pmatrix}0&1\\1&0\end{pmatrix}$. This satisfies all conditions. Do note that $(AB)^n=I$ if $n$ is a multiple of $4$, and $(AB)^n\ne I$ for all other $n$.
