Matrix representation of an operator relative to the given basis A linear operator A in the three-dimensional space is determined by the transformation of the coordinates of a vector $\begin{pmatrix}
                             x\\
                             y\\
z\\
                             \end{pmatrix}$
whose coordinates are given in the standard basis, this vector after transformation becomes 
$\begin{pmatrix}
x-2y\\
y+z\\
y-x\\
                             \end{pmatrix}$.
The new basis in the space is chosen as 
$\begin{pmatrix}
1\\
1\\
1\\
                             \end{pmatrix}$, 
$\begin{pmatrix}
1\\
1\\
0\\
                             \end{pmatrix}$,
$\begin{pmatrix}
1\\
0\\
0\\
                             \end{pmatrix}$.
Find the matrix representation of this operator relative to the given basis.
 A: If I understood right, $T(x,y, z)=(x-2y,y+z,y-x)$ thus $$T(1, 1, 1)=(1-2, 1+1, 1-1)=(-1, 2, 0)=0(1, 1, 1)+2(1,1,0)-3(1, 0, 0)$$
and $$T(1, 1, 0)=(-1,1,0)=0(1, 1, 1)+1(1,1,0)-2(1, 0, 0)$$
and $$T(1, 0, 0)=(1,0,-1)=-1(1, 1, 1)+1(1,1,0)+1(1, 0, 0)$$
\begin{bmatrix}
    0 & 0 & -1  \\
    2 & 1 & 1  \\
-3 & -2 & 1
  \end{bmatrix}
A: You need to check again what a matrix $M:V_1\to V_2$ represents. In particular for an endomorphism like in your case (meaning both vector spaces are the same $V_1=V_2$), each column $j$ of the matrix $M$ gives the image of the basis vector $b_i$ by that matrix, i.e., $Mb_i$.
Thus, in the standard basis 
$$M=\begin{pmatrix}1 & -2& 0\\0 & 1& 1\\-1 & 1& 0\end{pmatrix}$$
Let's call $v_1=(1,1,1)^T,\,v_2=(1,1,0)^T,\,v_3=(1,0,0)^T$. 
We have 
$$Mv_1=(-1,2,0)^T=2v_2-3v_3\\Mv_2=(-1,1,0)=v_2-2v_3\\Mv_3=(1,0,-1)=-v_1+v_2+v_3$$
Therefore, in the new basis 
$$M'=\begin{pmatrix}0 & 0& -1\\2 & 1& 1\\-3 & -2& 1\end{pmatrix}$$
Which is the answer you are seeking.
Notice, if we call $C$ the matrix providing the change of basis from the new $\{v_i\}$ to the old one  $\{e_i\}$, it is
$$C=\begin{pmatrix}1 & 1& 1\\1 & 1& 0\\1 & 0& 0\end{pmatrix}$$
and it holds that
$$M'=C^{-1}MC$$
You could have also proceeded by determining $C,\,C^{-1}$ and doing the above matrix multiplication. For a "small" matrix ($3\times 3$) like in the present case, however, it is simpler and more elegant to do as we did.
