(Update: Oh sorry; I'm new here. Didn't notice the much more substantial link to the mathoverflow. But perhaps it is interesting for some newbie anyway...)
If the function is a polynomial (or powerseries) with constant term $\ne0$ this is difficult, but sometimes a meaningful solution can be given.
If the function is as above, but the constant term is zero, then it is just the question of relatively simple manipulation of the formal powerseries/polynomial to construct a "compositional" (or "iterative") root.
There is a very good deal in L.Comtet "Advanced Combinatorics" at pages around 130..150 (don't have it around).
Also the keywords "Bell-matrix", "Carleman-matrix" are helpful: these matrices transform the problem of funtion composition/iteration to matrix-products/matrix-powers. (The matrix-notation just implies the required formal powerseries-manipulations) . This is well established for functions $f(x)= ax + bx^2 + cx^3 +\cdots$.
If the constant term does not vanish, $f(x) = K + ax + bx^2 + cx^3 +\cdots$, then things are usually much more difficult. But there is one -I think: again well established- workaround:
Rewrite $f(x)$ as some function $g(x+x_0) - x_0$ such that now $g(x)$ has no constant term and then apply the above mentioned procedures (solving for iterative square root and the like) to $g(x)$.
Example: denote the iterative root of a some function $f(x)$ by $f^{[1/2]}(x)$ then solve
$$f^{[1/2]}(x) = g^{[1/2]}(x-x_0)+x_0$$
where you first must find $x_0$.
For a very basic introduction you may try my small essay at
go.helms-net.de/math/tetdocs
and look for the article "continuous functional iteration".