Is There a Concept of Fractional Composition? Does there exist a concept of fractional composition for functions? Continuous or differentiable functions?
 A: Yes, fractional function iteration is defined.
Like it works for natural iteration
$$f^{(n)}(x):=f(f(\cdots f(x)\cdots)),$$
and (negative) integer iteration
$$f^{-n}(x):=(f^{-1})^{(n)}(x)$$
you can generalize to rationals with the functional equation
$$g(x):=f^{(n/m)}(x)\iff g^{(m)}(x)=f^{(n)}(x).$$
For example, the iterative square root is defined as
$$g(x):=f^{1/2}(x)\iff g(g(x))=f(x).$$
In the case of the identity function, you get the so-called Babbage equation,
$$g(g(x))=x.$$
A particular solution is $g(x)=-x$, and as you can check, more solutions are found as
$$g(x)=h^{-1}(-h(x))$$ where $h$ is an arbitrary invertible function.
For instance, with $h(x)=\ln(x+1)$,
$$g(x)=\exp(-\ln(x+1))-1=-\frac x{x+1}.$$
So this square root isn't unique.

If $f(x)=x^r$ is a power law, the $n^{th}$ iterate is $((x^r)^{r\cdots})^r=x^{r^n}$ so that a particular solution of 
$$g^{(m)}(x)=(x^r)^{(n)}$$ is another power law
$$g(x)=x^s$$ with $s^m=r^n$, or $s=r^{n/m}$.
$$(x^r)^{(q)}=x^{r^q}.$$

As another example, the iteration of a linear law gives
$$(ax+b)^{(n)}=a(\cdots a(ax+b)+b\cdots)+b=a^nx+\frac{a^n-1}{a-1}b,$$ so that a fractional iterate can also be linear
$$(ax+b)^{(n/m)}=g(x)=Ax+B$$ with
$$A^m=a^n,\frac{A^m-1}{A-1}B=\frac{a^n-1}{a-1}b$$
or
$$A=a^{n/m},B=b\frac{a^n-1}{a-1}\frac{A-1}{A^m-1}=\frac{a^{n/m}-1}{a-1}.$$
$$(ax+b)^{(q)}=a^qx+\frac{a^q-1}{a-1}b.$$
