find matrix of linear transformation in another basis I can't get the workflow.
I have matrix:
$$A=\begin{pmatrix}-1&1&2\\0&3&1\\5&-1&1 \end{pmatrix}$$
I have to find matrix of this transformation in tha basis of $\langle f_1,f_2,f_3 \rangle$ if $$f_1 =e_2+e_3 \\f_2=e_1\\f_3=e_1-e_3$$
I know that there's formula $$A'=T^{-1}AT$$
where I believe that $A$ is given matrix, but do not know what are $T$ and $T^{-1}$
 A: Let $\mathbf{x} = (x_{1}, x_{2}, x_{3})^T$ be the coordinates of a point in the $e$-basis and let $\mathbf{y} = (y_{1}, y_{2}, y_{3})^T$ be the coordinates of the same point in the $f$-basis.
It is the same point, so we require the following condition.
$$
x_{1} \mathbf{e}_{1} + 
x_{2} \mathbf{e}_{2} + 
x_{3} \mathbf{e}_{3} 
=
y_{1} \mathbf{f}_{1} +
y_{2} \mathbf{f}_{2} +
y_{3} \mathbf{f}_{3}
$$
The question gives the way of writing the $f$-basis vectors in terms of the $e$-basis vectors:
$$
\begin{aligned}
\mathbf{f}_1 &= \mathbf{e}_1 + \mathbf{e}_2 \\
\mathbf{f}_2 &= \mathbf{e}_2 \\
\mathbf{f}_3 &= \mathbf{e}_1 - \mathbf{e}_3
\end{aligned}
$$
We can substutite these formulas into the equation for the  coordinates above
$$
x_{1} \mathbf{e}_{1} + 
x_{2} \mathbf{e}_{2} + 
x_{3} \mathbf{e}_{3} 
=
y_{1} (\mathbf{e}_{1} + \mathbf{e}_{2}) +
y_{2} \mathbf{e}_{2} +
y_{3} (\mathbf{e}_{1} - \mathbf{e}_{3})
$$
$$
x_{1} \mathbf{e}_{1} + 
x_{2} \mathbf{e}_{2} + 
x_{3} \mathbf{e}_{3} 
=
y_{1} \mathbf{e}_{1} + y_{1} \mathbf{e}_{2} +
y_{2} \mathbf{e}_{2} +
y_{3} \mathbf{e}_{1} - y_{3} \mathbf{e}_{3}
$$
$$
x_{1} \mathbf{e}_{1} + 
x_{2} \mathbf{e}_{2} + 
x_{3} \mathbf{e}_{3} 
=
(y_{1} +  y_{3}) \mathbf{e}_{1} 
+
(y_{1}  + y_{2} ) \mathbf{e}_{2} 
- y_{3} \mathbf{e}_{3}
$$
Now $\mathbf{e}_1$, $\mathbf{e}_2$ and $\mathbf{e}_3$ are three linearly independent vectors so that the components of each vector can be equated on both sides of the above. I.e. we can write:
$$
\begin{aligned}
x_1 &= y_1 + y_3 \\
x_2 & =  y_1 + y_2 \\
x_3 &= -y_3
\end{aligned}
$$
The following is exactly the same as the above with slightly different spacing
$$
\begin{aligned}
x_1 &= y_1 & &+y_3 \\
x_2 & =  y_1 &+ y_2 & \\
x_3 &= & &-y_3
\end{aligned}
$$
Writing the equations that relate the coordinates in this way, we can see how the set can be written as a single matrix equation:
$$
\begin{pmatrix}
x_{1} \\ x_{2} \\ x_{3}
\end{pmatrix}
=
\begin{pmatrix}
1 & 0 & 1 \\
1 & 1 & 0 \\
0 & 0 & -1 
\end{pmatrix}
\begin{pmatrix}
y_{1} \\ y_{2} \\ y_{3}
\end{pmatrix}
\Rightarrow 
\mathbf{x}
=
\begin{pmatrix}
1 & 0 & 1 \\
1 & 1 & 0 \\
0 & 0 & -1 
\end{pmatrix}
\mathbf{y}$$
This shows how we can use a matrix to convert coordinates in the $f$-basis to coordinates in the $e$-basis, i.e. $\mathbf{x} = T \mathbf{y}$, i.e. it represents the matrix $T$ in the formula
$$
A' = T^{-1} A T
$$
where $A$ is the transformation that is applied to coordinates in the $e$-basis. The above formula is applied 
Having found $T$, we can find its inverse (by hand or with some software):
$$
T^{-1}
=
\begin{pmatrix}
 1& -1&  0 \\
 0&  1&  0 \\
 1& -1& -1
\end{pmatrix}
$$
and  finally, we can calculate $A'$
$$
A'
=
\begin{pmatrix}
-3& -2& -2 \\
3&  3& -1 \\
-7& -1& -6
\end{pmatrix}
$$
which is the matrix of the transformation that is applied to coordinates in the $f$-basis.

This next part goes into how the formula relating the two transformation matrices in the different bases is derived.
If we write $\mathbf{u}$
 for the result of applying $A$ to the $e$-basis vector $\mathbf{x}$ and if we write
$\mathbf{v}$ for the result of applying $A'$ to the $f$-basis vector $\mathbf{y}$.
$$
\begin{aligned}
\mathbf{u} &= A \mathbf{x} \\
\mathbf{v} &= A' \mathbf{y} \\
\end{aligned}
$$
The vectors 
$\mathbf{x}$ and $\mathbf{y}$
correspond to the same point in the two different bases and so do the pair of vectors
$\mathbf{u}$  and $\mathbf{v}$. In other words they can be written:
$$
\begin{aligned}
\mathbf{x} &= T \mathbf{y} \\
\mathbf{u} &= T \mathbf{v} \\
\end{aligned}
$$
This means we can write the following
$$
\begin{aligned}
\mathbf{u} &= T \mathbf{v} \\
A \mathbf{x} &= T \mathbf{v} \\
A \mathbf{x} &= T A' \mathbf{y} \\
A T\mathbf{y} &= T A' \mathbf{y} \\
T^{-1} A T\mathbf{y} &=  A' \mathbf{y} \\
\end{aligned}
$$
As this works for all $\mathbf{y}$ we can conclude that $T^{-1} A T  =  A' $.
A: $T$ is the matrix whose columns are $f_1, f_2, f_3$. That is:
$$T=\begin{pmatrix}2&-1&-3\\6&0&-1\\-2&5&4 \end{pmatrix}$$
You just have to compute the inverse and apply your formula.
