# Find the matrix A of the linear transformation T(M) I know that if I substitute the first matrix for $T(M)$ I see what T does to each of the basis vectors. I don't understand how that creates a $3\times 3$ matrix though.

I was looking at this question for a hint, but I don't understand it still. Find the matrix of the given linear transformation $T$.

You decompose your matrix $M=\begin{pmatrix}a&b\\0&c\end{pmatrix}$ and decompose it as $M=a\begin{pmatrix}1&0\\0&0\end{pmatrix}+b\begin{pmatrix}0&1\\0&0\end{pmatrix}+c\begin{pmatrix}0&0\\0&1\end{pmatrix}=aE_1+bE_2+cE_3$ where $(E_1, E_2, E_3)$ is a basis for the space of upper triangular matrices.
Now you have another matrix $A=\begin{pmatrix}5&2\\0&7\end{pmatrix}$ which is upper triangular so $AM$ will also be upper triangular. So now you want to know how to express $AM$ as a linear combinaison of $E_1, E_2$ and $E_3$. You use $AM=A(aE_1+bE_2+cE_3)=aAE_1+bAE_2+cAE_3$. Now you compute each one of the $AE_i$, express it as a linear combinaison of $E_1,E_2$ and $E_3$ and then rearange the terms and you'll get $AM=f(a,b,c)E_1+g(a,b,c)E_2+h(a,b,c)E_3$ where $f,g,h \in \mathcal L\left(\Bbb R^3,\Bbb R\right)$. Given that, you should be able tocompute the matrix you want.
The matrix of a transformation $T$ in a basis $(e_i)_i$ can be written with column vectors $[..T(e_i)..]$, all coordinated in the given basis.
In this particular case, you should calculate the $3$ matrix products: $T(e_1),\ T(e_2),\ T(e_3)$, then express these three results as the linear combination of $e_1,e_2,e_3$ and write the coefficients in the columns.
For example, $T(e_3)=T\pmatrix{0&0\\0&1}=\pmatrix{0&2\\0&7}=7e_3+2e_2-2e_1=-2e_1+2e_2+7e_3$.