It is a linear operator on $\Bbb R[X]$, not in that space. And for linear operators, the multiplication structure of polynomials is ignored, so it cannot be a). Option b) is better, but is incomplete. First of all you need to say this condition $U(T(f))=f$ should hold for any $f\in\Bbb R[X]$ (in spite of appearances, $f$ is not a function but a polynomial). But even then this is only half of the requirement. One should also have $T(U(p))=p$ for every $p\in\Bbb R[X]$. Unlike in finite dimension, that second condition is not a consequence of the first. For instance, if $T$ is the operation of multiplying by some fixed polynomial, say by $2X^3-X+7$, and $U$ is the operation of taking the Euclidean quotient by that polynomial (ignoring any remainder in the division), then both are linear operations and $U(T(p))=p$ holds for any polynomial $p$, but $T(U(p))$ does not: it only holds when the Euclidean division in computing $U(p)$ produces no remainder.
And you cannot have (finite) matrices because the space has infinite dimension.