Linear operator and its corresponding matrix. 
There's linear operator $A: \mathbb{R}_2[x] \to \mathbb{R}_2[x]$ defined as $(A(p))(x):=p'(x+1)$.
Find all possible values for $a, b, c \in \mathbb{R}$ for which matrix $\begin{bmatrix} a & 1 & 0 \\ b & 0 & 1 \\ c & 0 & 0 \end{bmatrix} $ can be matrix of this linear operator with respect to some basis of space $\mathbb{R}_2[x]$.

Let's take arbitrary element of $\mathbb{R}_2[x]$, that would be polynomial $p(x)=dx^2 + ex + f$, where $d, e, f \in \mathbb{R}$. Now, $p'=2dx + e$ and $p'(x+1)=2dx^2 + (2d + e)x +e$ which means that this operator actually does following mapping $(d,e,f) \mapsto (2d,2d+e,e)$. but looking at matrix, there's no such ordered pair of three numbers that satisfies $(d,e,f)\mapsto (1, 0 , 0)$. What am i doing wrong here, and what is the correct way to solve this?
 A: Recall that the columns of the matrix of a transformation relative to some basis are the images of the basis vectors. Let $(p_1,p_2,p_3)$ be a basis for which the matrix of $A$ takes the given form. The third column tells us that $Ap_3=p_2$ and the second column that $Ap_2=p_1$, so $p_1=AAp_3$. The first column says that $Ap_1=ap_1+bp_2+cp_3$. Expanding this in terms of $p_3$ and using the fact that $p_3$, $Ap_3$ and $AAp_3$ must be linearly independent should let you find the possible first columns of the matrix. Note that, since $A$ never increases degree, $p_3$ must be second-degree.  
Update: Another approach is to examine the eigenvalues of the given matrix. Its characteristic polynomial is easily seen to be $\lambda^3-a\lambda^2-b\lambda-c$, which means that we have $$\begin{align}a&=\lambda_1+\lambda_2+\lambda_3 \\ b&=-(\lambda_1\lambda_2+\lambda_1\lambda_3+\lambda_2\lambda_3) \\ c&=\lambda_1\lambda_2\lambda_3.\end{align}$$ $A$ has a non-trivial kernel (any constant polynomial is mapped to zero), so zero is an eigenvalue, which means that $c=0$, leaving $$\begin{align}a&=\lambda_1+\lambda_2\\b&=-\lambda_1\lambda_2\end{align}$$ where $\lambda_1$ and $\lambda_2$ are the other eigenvalues (also possibly zero) of $A$. These eigenvalues can be found by, e.g., computing the matrix of $A$ relative to the standard basis. While this approach doesn’t demonstrate a basis for which the matrix has this shape, it does immediately show that there’s only one possible set of values for $a$, $b$ and $c$.
A: The choice of the base which is expressed this operator does not
fit, does not give the shape of the desired matrix.
so let $P_1=3+7X+4X^2$, $P_2=1+3X+2X^2$, $P_3=1+X+X^2$ and
$B=\{P_1,P_2,P_3\}$, then  the matrix of this operator in this
basis B is
$\left(
\begin{array}{ccc}
3 & 1 & 0 \\
-2 & 0 & 1 \\
0 & 0 & 0
\end{array}
\right)$
A: The matrix of your operator (which I'll rename to $T$ in order not to get confused with matrices) with respect to the basis $\mathcal{B} = (1,x,x^2)$ is given by
$$ [T]_{\mathcal{B}} = A = \begin{pmatrix} 0 & 1 & 2 \\ 0 & 0 & 2 \\ 0 & 0 & 0 \end{pmatrix}. $$
Call your other matrix $B$. The operator $T$ will be represented by $B$ if and only if the matrix $B$ is similar to $A$. The characteristic polynomial of $A$ is $x^3$ (as $A$ is nilpotent). The characteristic polynomial of $B$ is
$$ \det(xI - A) = \det \begin{pmatrix} x - a & -1 & 0 \\ -b & x & -1 \\ -c & 0 & x \end{pmatrix} = (x-a)x^2 +(-bx - c) = x^3 - ax^2 - bx - c. $$
Since similar matrices have the same characteristic polynomial, we must have $a = b = c = 0$. Finally, you can check that indeed
$$ A = \begin{pmatrix} 0 & 1 & 2 \\ 0 & 0 & 2 \\ 0 & 0 & 0 \end{pmatrix}, \,\,\, B = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix} $$
are similar. An explicit basis $\mathcal{C}$ with respect to which $[T]_{\mathcal{C}} = B$ is given by $\mathcal{C} = \left(1, x, \frac{x^2}{2} \right)$.
Actually, you can do this without any calculations. The operator $T$ is readily seen to be nilpotent and so $a = 0$ (as we must have $\operatorname{trace}(B) = a = 0$). It has one dimensional kernel which immediately implies that $b = c = 0$.

The solution above assumed that $T(x) = p'(x+1)$. If, instead, $T(x) = p'(x) \cdot (x + 1)$ then
$$ [T]_{\mathcal{B}} = \begin{pmatrix} 0  & 1 & 0 \\ 0 & 1 & 2 \\ 0 & 0 & 2 \end{pmatrix}. $$
The matrix $[T]_{\mathcal{B}}$ is upper triangular and so its eigenvalues are $\lambda = 0,1,2$. By comparing characteristic polynomials we see that
$$ x(x-1)(x-2) = (x^2 - x)(x-2) = x^3 - 3x^2 + 2x = x^3 - ax^2 - bx - c $$
so $a = 3, b = -2, c = 0$ and 
$$ B = \begin{pmatrix} 3 & 1 & 0 \\ -2 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}. $$
The matrices $A$ and $B$ are similar and so there must exist a basis $\mathcal{C}$ such that $[T]_{\mathcal{C}} = B$. If $P^{-1}BP = A$ then the columns of $P$ represent the basis $\mathcal{C}$ with respect to the basis $\mathcal{B}$ (that is, $P = [\operatorname{id}]_{\mathcal{C}}^{\mathcal{B}}$).
