Representation of $\mathrm{Diff}(M)$ onto $C^\infty(M)$ I came up with the following question when studying symmetries in physics. Please note that some part of my question is not precise.
Let $M$ be a smooth manifold, and let $\mathrm{Diff}(M)$ be the diffeomorphism group of $M$. Consider $\mathrm{Diff}(M)$ as an (infinite-dimensional) Lie group. Then, we have a natural Lie group representation of $\mathrm{Diff}(M)$ on the space of smooth functions $C^\infty(M)$ by
$$\rho: \mathrm{Diff}(M) \to \mathrm{GL}(C^\infty(M)),$$
where $\rho(F)(f) = f \circ F^{-1}$. In other words, a diffeomorphism $F:M \to M$ is regarded as a coordinate transform. Here, we have to consider the inverse $F^{-1}$ in the definition of $\rho$ to ensure that $\rho$ is a representation, i.e., $\rho(F\circ G) = \rho(F) \circ \rho(G)$.
Question: Lie group representation induces the corresponding Lie algebra representation. What is the Lie algebra representation corresponding to $\rho$?
Some thoughts: First, we should determine what is the Lie algebra of $\mathrm{Diff}(M)$. I suspect that it is $\mathfrak X(M)$, the space of vector fields on $M$, since vector field encodes information of the infinitesimal transformation. Then, for $X\in \mathfrak X(M)$, considering (formally) $e^{tX} \in \mathrm{Diff}(M)$, where $t\in\mathbb R$ is infinitesimal, we write
$$(e^{tX} \cdot f)(x) := \rho(e^{tX})(f)(x) = f(e^{-tX}x),$$
where we used the simplified notation $e^{tX} \cdot f$ to denote the representation $\rho$. By Taylor expanding both sides and considering the terms up to $O(t)$, we have
$$f(x) + t (X\cdot f)(x) =  f(x^a - tX^a) = f(x) - t X^a \partial_a f(x).$$
Hence, I expect
$$X\cdot f = -X^a \partial_a f.$$
However, the minus sign here is somewhat strange. Is this a Lie algebra representation? (If there were no minus sign, that the above is indeed the Lie algebra representation.) I cannot figure out where is wrong.
 A: The corresponding Lie algebra representation "ought to be" the action of vector fields by the Lie derivative. The sign is supposed to be there. The issue, as far as I can tell, is that there is a sign convention in the definition of the Lie bracket on vector fields and to make it consistent with the conventions given here that sign should be reversed; that is, the definition of the Lie bracket on vector fields should be the negative of the commutator bracket.
Basically I think this comes down to whether you think a vector field $X$ should exponentiate to $f(x) \mapsto f(e^{tX} x)$ (the convention that defines the usual Lie derivative and the usual Lie bracket on vector fields) or $f(x) \mapsto f(e^{-tX} x)$ (the convention you are adopting here).
A: A few comments to expand on Qiaochu Yuan's answer: it is useful to distinguish between right group actions and left group actions. In your case, you are looking at a left action of the group of diffeomorphisms on the space of functions.
Consider a Lie group $G$, and let $\mathfrak{g}:=\operatorname{Lie}(G)$ be its Lie algebra; recall that this $\mathfrak{g}=T_eG$ is identified with the space of left-invariant vector fields on $G$.
For a left-action of $G$ on a manifold $M$, let's say $x\to\lambda_g(x)$, one can define an infinitesimal action of $\mathfrak{g}$ on $M$ as follows. To each $a\in\mathfrak{g}=T_eG$ we associate the vector field $$\hat{a}_x:=\partial_{t=0}\lambda_{\operatorname{exp(t\,a)}}(x).$$
This is a Lie algebra anti-homomorphism between $\mathfrak{g}$ and the set $\mathcal{X}(M)$ of vector fields on $M$: to get a Lie algebra homomorphism instead, you should define $\hat{a}$ as $$\hat{a}_x:=\partial_{t=0}\lambda_{\operatorname{exp(-t\,a)}}(x),$$ see for example Proposition $3.8$ in Appendix 5 of this book.
From this point of view then it is not surprising that you have a minus sign in your infinitesimal action. This can be corrected, if you need to do so, by either considering a right-action of $G$, or by changing the definition of Lie bracket by a minus sign (which is roughly the same as considering $\mathfrak{g}$ as the set of right-invariant vector fields).
