The trace and the determinant are the most useful invariants because the trace is additive and the determinant is multiplicative. The other coefficients of the characteristic polynomial are neither. The determinant also has a clear geometric interpretation. In addition, all of the coefficients of the characteristic polynomial of an operator $T$ can be computed from the traces of the operators $T^n$; this is one reason why it is not so surprising that traces of group elements in group representations carry a lot of information.
This is not to say that people never use the other invariants, although they don't tend to have special names. For example, my understanding is that the Killing form in Lie theory, an important tool, was discovered by messing around with characteristic polynomials. And the construction underlying the coefficients of the characteristic polynomial, the exterior algebra, is enormously useful.