I read everywhere :"By definition, the characteristic classes of smooth manifolds are invariant under diffeomorphisms." Does it follow from de Rham cohomology? If this is so, then what about topological manifolds? In this case we cannot use diffeomorphism invariance of de Rham cohomology. Besides even if we know that cohomology classes are preserved under diffeomorphisms (is this correct for the cohomology with all coefficients (not only Z, but also Q and R coefficients)?), how can we be sure that these classes are also preserved? I cannot see it from the definitions.
To say it crisply, characteristic classes are invariant under diffeomorphisms because they are invariant under homeomorphisms!
A little more in detail, characteristic classes for real vector bundles (for example) are defined for paracompact topological spaces , which needn't even be topological manifolds .
Now if you have chosen a ring of coefficients $\Lambda$ and an element $c\in H^*(G_r(\mathbb R^\infty),\Lambda$), the pull-back $f^*(c)\in H^*(X,\Lambda)$ will get you the corresponding characteristic class of $E$.
There is no need for differentiability in this theory, but of course if $X$ happens to be a manifold you'll get a whole bunch of new bundles derived from the tangent bundles and characteristic classes will tell you a lot about them.
You can only talk meaningfully about diffeomorphisms on smooth manifolds, since only they guarantee smooth transition functions between patches. In that case, de Rham's Theorem guarantees an isomorphism between de Rham cohomology and singular cohomology.