Trace and identity are the only linear matrix invariants?

This question is obviously related to that recent question of mine, but I feel it’s sufficiently different to be posted as a separate question. Let $V$ be a finite-dimensional space. Let ${\cal L}(V)$ denote the space of all endomorphisms of $V$. Say that an endomorphism $\phi$ of ${\cal L}(V)$ is invariant when it satisfies $$\phi (gfg^{-1})=g\phi(f)g^{-1}$$ for any $f,g \in {\cal L}(V)$ with $g$ invertible.

Prove or find a counterexample or provide a reference : $\phi$ is invariant iff there are two constants $a,b$ such that $\phi(f)=af+b{\sf tr}(f){\bf id}_V$ for all $f$. With the help of a PARI-GP program, I have checked that this is true when ${\sf dim}(V) \leq 5$. Intuitively, the similitude invariants of a matrix are functions of the coefficients of the characteristic polynomial, and the second largest coefficient, the trace, is the only linear one.

-

Let $G=\operatorname{GL}(V)$ be the group of automorphisms of $V$. Then as a $G$-module, $\mathcal L(V)$ is isomorphic to $V\otimes V^*$, and $\hom_k(\mathcal L(V),\mathcal L(V))$ is isomorphic to $V^{\otimes 2}\otimes V^{*\otimes 2}$. Your question is, in this language:
what is the dimension of the $G$-invariant subspace of $V^{\otimes 2}\otimes V^{*\otimes 2}$?
Notice that $V^{\otimes 2}\otimes V^{*\otimes 2}$ is isomorphic as a $G$-module to $hom_k(V^{\otimes 2},V^{\otimes 2})$, so its invariant subspace is actually the space of $G$-equivariant maps, $hom_G(V^{\otimes 2},V^{\otimes 2})$. The fact that there are exactly two linear invariants in your sense is the case $m=2$ of the claim stated on page 31 of those notes —this is part of what's called Schur-Weyl duality.