Not the best way, but you could also try diagonalisation. The caveat with diagonalisation is that for certain values of $a$ and $b$ (in particular, if $a = 1$ and $b \neq 0$), the matrix won't be diagonalisable. However, if we make the assumption that $a \neq 1$, then we should end the process with a perfectly valid expression for $A^n$ that will work for all $a \neq 1$, and by continuity, we can conclude that it works for $a = 1$ too (or take the formula, and prove by induction).
Also, I know you want hints, so I hid everything behind spoiler boxes.
The eigenvalues are $1$ and $a$, and we will assume they are different. We have,
$$A - I = \begin{pmatrix} a - 1 & b \\ 0 & 0 \end{pmatrix},$$
with eigenvector $(-b, a - 1)$. Also,
$$A - aI = \begin{pmatrix} 0 & b \\ 0 & 1 - a \end{pmatrix},$$
with eigenvector $(1, 0)$. So, let
$$P = \begin{pmatrix} 1 & -b \\ 0 & a - 1\end{pmatrix},$$
giving us
$$P^{-1} = \frac{1}{a - 1}\begin{pmatrix} a - 1 & b \\ 0 & 1\end{pmatrix}.$$
We should have
$$A = P\begin{pmatrix} a & 0 \\ 0 & 1\end{pmatrix}P^{-1},$$
so
$$\begin{align*} A^n &= P\begin{pmatrix} a^n & 0 \\ 0 & 1\end{pmatrix}P^{-1} \\ &= \frac{1}{a - 1}\begin{pmatrix} 1 & -b \\ 0 & a - 1\end{pmatrix}\begin{pmatrix} a^n & 0 \\ 0 & 1\end{pmatrix}\begin{pmatrix} a - 1 & b \\ 0 & 1\end{pmatrix} \\ &= \frac{1}{a - 1}\begin{pmatrix} 1 & -b \\ 0 & a - 1\end{pmatrix}\begin{pmatrix} a^{n+1} - a^n & ba^n \\ 0 & 1\end{pmatrix} \\ &= \frac{1}{a - 1}\begin{pmatrix} a^{n+1} - a^n & ba^n - b \\ 0 & a - 1\end{pmatrix} \\ &= \begin{pmatrix} a^n & b \frac{a^n - 1}{a - 1} \\ 0 & 1\end{pmatrix} \\ &= \begin{pmatrix} a^n & b(1 + a + a^2 + \ldots + a^{n-1}) \\ 0 & 1\end{pmatrix}.\end{align*}$$
That last formula must hold (at least) for $a \neq 1$, but due to the continuity of matrix powers, it must also hold at $a = 1$ too.