Real function $f$ such that $f(f(x)) = x^3 - 3x^2 + 3x$ Does there exist a function $f:\mathbb{R} \to \mathbb{R}$ such that $f(f(x)) = x^3 - 3x^2 + 3x$?
Since $f(x) = f(y)$ implies $f(f(x)) = f(f(y))$ and so $(x-1)^3 - 1 = (y-1)^3 - 1$, i.e. $x=y$, we have that $f$ must be injective. In particular, $f(f(0)) = 0$ and $f(z) \neq 0$ for $z\neq f(0)$.
I was thinking to try to mimic a known nice construction for $f(f(x)) = ax$ for $a > 0$ - namely $f(x) = -x$ if $x\geq 0$ and $f(x) = -ax$ if $x<0$. But if we try to use it here e.g. like $f(x) = -x$ for $x\geq 0$ and $f(x) = -x(x^2+3x+3)$ since $g(x) = x^2 + 3x + 3$, even though positive, does not satisfy $g(x) = g(-x)$.
Any idea how to fix this or for perhaps some different construction? (If it is "easy to describe" it would be awesome, but I am open to any suggestions.) Any help appreciated!
 A: It is quite clear that there is no polynomial solution to this. But in fact there is a solution. By differentiating you’ll see that if $g(x)=x^3-3x^2+3x$ then $g'>0$ almost everywhere, thus $g$ is strictly increasing. This means that $g$ is bijective.
So now let’s take
$$ f(g(x))=f(x^3-3x^2+3x)=f(f(f(x))) = f(x)^3-3f(x)^2+3f(x)=g(f(x)) $$
As $g(x)=x$ has exactly three solutions $x=0,1,2$ we get that for these three values we need to have
$$ f(x) = f(x)^3-3f(x)^2+3f(x) $$
so $f(x)$ is also a solution to $f(x) = g(f(x))$. As $g$ is bijective there are no further solutions to $g^k(x)=g^l(x)$ and thus any other orbit $g^n(x)$, $n\in\mathbb Z$ is infinite in both directions.
Also since $g$ is bijective then as long as $y$ is not in the orbit of $x$ then the orbits are disjoint (else $g(y)^k=g(x)^l$, so $g(y)=g(x)^{l-k}$).
Note that as $f$ needs to be injective it must not map two different orbits into the same orbit.
Thus if we partition $\mathbb R$ in disjoint orbits $g^n(x)$ and if we define two representant system $S,R$ of these orbits, we can define $f(x)$ on $S$ as any bijection onto $R$. The further values of $f$ are then given by $f(g(x))=g(f(x))$ (and thus also $g^{-1}(f(g(x)) = f(x)$ and thus $f(g^{-1}(x))=g^{-1}(f(x))$). The resulting function is clearly bijective.
Note that if for some $s\in S$ we had $f(s)= g^k(s)$ then $f(f(s))=f(g^k(s)) = g^kf(s) = g^{2k}(s)$, so this will not solve the original problem (unless $s$ is a solution to $x=g(x)$, as then $f(f(s)) = s = g(s)$).
Now how can we solve the original problem? Suppose $s,r\in S,R$ so that $f(s) = r$. Then we need
$$ f(f(s)) = f(r) = g(s)$$
Thus if we choose $S$ so that $r\in S$ then $g(s)\in R$.
By the well-ordering principle we can guarantee that $S$ can be arranged in pairs $(s,t)$. So we define $f(s)=t$ and $f(t) = g(s)$.
Then this implies that
$$ f(f(s)) = g(s)$$
and
$$ f(f(t)) = f(g(s)) = g(f(s)) = g(t) $$
Note that the three solutions to $x=g(x)$ must map to each other and cannot be arranged in pairs. This enforces that for these $x$ we have $f(x)=x$.
Then we get:
$$f(f(x)) = x = g(x)$$
for these $x$ and for all other $x$ with $x=g^k(s)$
$$ f(f(x))=f(f(g^k(s))) =g^k(f(f(s))) = g^k(g(s)) = g^{k+1}(s) = g(x)$$
A: Take the function
$$
f(x)=
\begin{cases}
(x-1)^{\sqrt 3}+1,\qquad x\geq 1,\\
1-(1-x)^{\sqrt 3},\qquad x< 1.
\end{cases}
$$
Then for $x\geq 1$ we have
$$ 
f(f(x))=\left((x-1)^{\sqrt 3}+1-1\right)^{\sqrt 3}+1=\left((x-1)^{\sqrt 3}\right)^{\sqrt 3}+1=(x-1)^3+1=x^3-3x^2+3x.
$$
If $x<1$ we get
$$ 
f(f(x))=1-\left(1-\left(1-(1-x)^{\sqrt 3}\right)\right)^{\sqrt 3}=1-\left((1-x)^{\sqrt 3}\right)^{\sqrt 3}=1-(1-x)^3=x^3-3x^2+3x.
$$
A: We can start by factoring it as,
$$f(f(x))=(x-1)^3+1$$
Since we notice that we have the function $h(x)=x+1$ and $h^{-1}(x)=x-1$ we can conjugate $f$ to make a new function which is easier to think about,
$$g(x)=h^{-1}(f(h(x)))$$
$$g(g(x))=h^{-1}(f(h(h^{-1}(f(h(x))))))=h^{-1}(f(f(h(x)))) = x^3$$
Now as Will Jagy alludes to in his comment on the question, we have the solution,
$$g(x)=\mathrm{sgn}(x)|x|^{\sqrt{3}}$$
The sign of x is necessary to be separated out because there's no clear way to define $\sqrt{3}$ power on negative numbers. Let's check that this works,
$$g(g(x))=\mathrm{sgn}(\mathrm{sgn}(x)|x|^{\sqrt{3}})\left|\mathrm{sgn}(x)|x|^{\sqrt{3}}\right|^{\sqrt{3}}$$
$$=\mathrm{sgn}(\mathrm{sgn}(x))\left||x|^{\sqrt{3}}\right|^{\sqrt{3}}$$
$$=\mathrm{sgn}(x)\left|x^{\sqrt{3}}\right|^{\sqrt{3}}$$
$$=\mathrm{sgn}(x)^3|x|^3$$
$$=x^3$$
Now we can conjugate back to get our answer,
$$f(x)=h(g(h^{-1}(x))) = \mathrm{sgn}(x-1)|x-1|^{\sqrt{3}}+1$$
