Formula for n-th composition of Mobius function. Given function: $f(x)=\frac{1}{x-1}       $
What is formula for $f_n$, if $f_2=f \circ f$  and $f_n=f\circ f_{n-1}$ for $n>2$?
I have one observation: $f_n=\frac{a}{b}$ where $a$ is denominator from $f_{n-1}$ and $b$ is some binomial.
 A: Yes, this is possible! This is a particular instance of a Mobius transformation and all such transformations have an easy form for their powers by using the idea of conjugacy. In particular, there is some other function
$$g(z)=\frac{az+b}{cz+d}$$
such that the expression $g(f(g^{-1}(z))$ simplifies to either $kz$ or $z+k$ for some $k$. Then, $g\circ f \circ g^{-1}$ can be iterated easily, which allows for a formula for $f^n$ as I will derive below.
In particular, let $\alpha=\frac{1+\sqrt{5}}2$ and $\beta=\frac{1-\sqrt{5}}2$ be the solutions to $f(z)=z$. Since we hope that $g(f(g^{-1}(z)))=kz$ fo some $z$, we wish to choose $g$ so that $g(\alpha)=0$ and $g(\beta)=\infty$ (i.e. $g$ divides by $0$ there - this is projective infinity). We choose
$$g(z)=\frac{z-\alpha}{z-\beta}$$
One may compute the inverse of $g$ to be
$$g^{-1}(z)=\frac{\beta z - \alpha}{z-1}.$$
One then computes that, if we let $k=-\frac{3+\sqrt{5}}2$ we have
$$g(f(g^{-1}(z)))=k\cdot z.$$
If we let $h(z)=kz$ then we find $g\circ f \circ g^{-1} = h$. Iterating this $n$ times and cancelling instances of $g^{-1}\circ g$ gives
$$g\circ f^n \circ g^{-1} = h^n.$$
Rearranging gives
$$f^n=g^{-1}\circ h^n \circ g$$
which, since $h^n(z)=k^n\cdot z$ is easy to compute, gives a closed form for $f^n$, if I've done the algebra correctly, where $\alpha,\beta,k$ are the previously defined constants:
$$f^n(z) = \frac{k^n-1+(\beta k^n - \alpha)z}{\beta - \alpha k^n + (k^n-1)z}$$

You can also show that
$$f^n(z)=\frac{F_{n-1}-F_{n-2}z}{F_n z - F_{n-1}}$$
where $F_n$ is the Fibonacci sequence with $F_0=F_1=1$. This can be shown easily via induction and more conceptually via the matrix representation of Mobius transformations.
