The characteristic polynomial is $(2-\lambda)(1-\lambda)-4=2-\lambda -2\lambda +\lambda^2 -4=\lambda^2-3\lambda -2$.
CH ensures you that
$$
A^2 - 3A = +2I
$$
hence
$$
A(A-3I)=2I \qquad (A-3I)A = 2I
$$
so that $A^{-1}=\frac{1}{2}(A-3I)$. From the equation above we also have, as you found,
$$
A^3 = 3A^2 +2A = 3(3A+2I) + 2A= 11A+6I.
$$
All in all,
$$
A^{-3} = (A^{-1})^3 = \frac{1}{8}(A-3I)^3 = 8^{-1}(A^3 - 9A^2 + 27A - 27I)
$$
which can be further simplified into
$$
= 8^{-1}(11A+6I - 9(3A+2I)+27A-27I) =
$$
$$
= \frac{1}{8}(11A+6I-27A -18 I +27A - 27I) = \frac{1}{8}(11A -39I)
$$
thus
$$
\boxed{A^{-3} = \frac{1}{8}\begin{bmatrix}
-17 & 44 \\
11 & -28
\end{bmatrix}}.
$$