From $x=r \cos{t}$, $y=\sin{t}$:
$$x'=(\cos{t}) r' - r (\sin{t}) \, t'$$
$$y'=(\sin{t}) r' + r (\cos{t}) \, t'$$
So we get
$$\left ( \begin{array}\\ \cos{t} & -r \sin{t} \\ \sin{t} & r \cos{t} \end{array} \right ) \left ( \begin{array}\\ r' \\ t' \end{array} \right ) = \left ( \begin{array}\\ r \cos{t} - r \sin{t} - r^3 \cos{t} + \sin{t} \cos{t} \\ r \cos{t} + r \sin{t} - r^3 \sin{t} - \cos^2{t} \end{array} \right ) $$
Multiply both sides by the matrix inverse to get
$$\left ( \begin{array}\\ r' \\ t' \end{array} \right ) = \frac{1}{r} \left ( \begin{array}\\ r\cos{t} & r\sin{t} \\ - \sin{t} & \cos{t} \end{array} \right ) \left ( \begin{array}\\ r \cos{t} - r \sin{t} - r^3 \cos{t} + \sin{t} \cos{t} \\ r \cos{t} + r \sin{t} - r^3 \sin{t} - \cos^2{t} \end{array} \right ) $$
Just do out the multiplication. It is messy, but there is a lot of cancellation and we get
$$\left ( \begin{array}\\ r' \\ t' \end{array} \right ) = \left ( \begin{array}\\ r-r^3 \\ 1 - \cos{t} \end{array} \right ) $$