How to solve the functional equation $ f(f(x))=ax^2+bx+c $ Find all real numbers $a,b,c\in\mathbb{R}$ for which there exists a function $f:\mathbb{R}\to\mathbb{R}$ such that:
$$
f(f(x))=ax^2+bx+c
$$
for all $x\in\mathbb{R}$.
The only thing I could deduce is:
$$
f(ax^2+bx+c)=af(x)^2+bf(x)+c
$$
Which doesn't help much. How to tackle the problem?
 A: SHORT ANSWER: A general solution to this problem is not known in the closed form, but some special cases can be solved. Sorry.
LONG ANSWER: Notice that if a function $f$ satisfies
$$f(x)=(g^{-1} \circ h \circ g)(x)$$
for some $g,h$, then
$$f^n(x)=(g^{-1} \circ h^n \circ g)(x)$$
where the superscript denotes functional iteration rather than exponentiation.
We can solve this equation for a whole class of quadratics
$$f^2(x)=q(x)=ax^2+bx+c$$
for which
$$c=\frac{b^2-2b}{4a}$$
This is because we can rewrite
$$q(x)=ax^2+bx+\frac{b^2-2b}{4a}$$
as
$$q(x)=a\bigg(x+\frac{b}{2a}\bigg)^2-\frac{b}{2a}$$
and so, by letting $g(x)=x-\frac{b}{2a}$ and $h(x)=ax^2$,
$$q(x)=(g^{-1} \circ h \circ g)(x)$$
and thus
$$q^n(x)=(g^{-1} \circ h^n \circ g)(x)$$
and since the formula for $h^n$ is
$$h^n(x)=a^{2^n-1}x^{2^n}$$
we have
$$q^n(x)=a^{2^n-1}\bigg(x+\frac{b}{2a}\bigg)^{2^n}-\frac{b}{2a}$$
and, finally,
$$f(x)=q^{1/2}(x)=a^{\sqrt 2-1}\bigg(x+\frac{b}{2a}\bigg)^{\sqrt 2}-\frac{b}{2a}$$
So there's the solution for that special case. There is another special case when 
$$c=\frac{b^2-2b-8}{4a}$$
but the solution to that case is much longer, and so I will omit it and leave it to you for independent research.
There is another special case involving trigonometric functions. Note that if we let
$$g(x)=\arccos x$$
$$h(x)=2x$$
we have
$$(g^{-1}\circ h\circ g)(x)=\cos(2\arccos x)$$
and, using the double-angle formula,
$$(g^{-1}\circ h\circ g)(x)=\cos^2(\arccos x)-\sin^2(\arccos x)$$
$$(g^{-1}\circ h\circ g)(x)=x^2-(1-x^2)$$
$$(g^{-1}\circ h\circ g)(x)=2x^2-1$$
and so $g^{-1}\circ h\circ g$ is a quadratic. Now let
$$q(x)=(g^{-1}\circ h\circ g)(x)$$
so that
$$q^n(x)=(g^{-1}\circ h^n\circ g)(x)$$
Now, since 
$$h^n(x)=2^n x$$
we have
$$q^n(x)=\cos(2^n\arccos x)$$
and
$$f(x)=q^{1/2}(x)=\cos(\sqrt{2}\arccos x)$$
which solves yet another special case. However, notice that this is only solved on the domain $[-1,1]$, because that is where $\arccos x$ is defined.
I am sure there are other special cases involving trigonometric functions, but I will leave those for you to find.
A: Here are some reflections for continuous and differentiable $f$. It turns out that there is a unique solution if some conditions are satisfied (you can look at the end for the final answer).



*

*$$
f(f(x))=ax^2+bx+c\implies f(ax^2+bx+c)=af(x)^2+bf(x)+c
$$

*If $f(x)=f(y)$, $x\neq y$ then:
$$
a(x+y)=b.
$$
If $a(x+y)=b$ then in any case $ff(x)=ff(y)$ which means that either $f(x)=f(y)$ or $a(f(x)+f(y))=b$. In any case there are two non-equal numbers $x'$ and $y'$ such that $f(x')=f(y')$ and they lie on two different sides of $\frac{-b}{2a}$. 

*If $f(x)=x$ then: $f(x)=ax^2+bx+c$ hence:
$$
ax^2+(b-1)x+c=0\implies (b-1)^2\geq 4ac; x=\frac{-b+1\pm\sqrt{(b-1)^2-4ac} }{2a}.
$$

*$f'(ax^2+bx+c)(2ax+b)=f'(x)(2af(x)+b)$ hence:
$$
x=\frac{-b}{2a}\implies f'(\frac{-b}{2a})=0\text{ or } f(\frac{-b}{2a})=\frac{-b}{2a}$$
On the other hand:
$$
f'(f(x)).f'(x)=2ax+b.
$$
So:


If $f'(x)=0$ then $x=\frac{-b}{2a}$. 

Note that from large enough $x$ and $y$ satisfying $a(x+y)=b$ and $f(x)=f(y)$


*

*If $f(\frac{-b}{2a})=\frac{-b}{2a}$, then $f\circ f(\frac{-b}{2a})=\frac{-b}{2a}$ which means:
$$
c-\frac{b^2}{4a}=-\frac b{2a}\implies b^2-4ac=2b.
$$


moreover:
$$
f'(f(\frac{-b}{2a})).f'(\frac{-b}{2a})=\implies f'(\frac{-b}{2a})=0.
$$
If $b^2-4ac\neq 2b$, $f'(\frac{-b}{2a})=0$; so in any case:

$$
f'(\frac{-b}{2a})=0.
$$

This means that the function is strictly increasing on the one side of $\frac{-b}{2a}$ and strictly decreasing on the other side.
Moreover if $f'(f(x))=0$ then $x=\frac{-b}{2a}$; so if there is $x$ such that $f(x)=\frac{-b}{2a}$ then $f'(f(x))=0$ which means that $x=\frac{-b}{2a}$. So:

$$f(\frac{-b}{2a})=\frac{-b}{2a}\implies b^2-4ac=2b$$

Without this condition there will be no answer.


*

*But if $a(x+y)=b$ and $f(x)\neq f(y)$, then $a(f(x)+f(y))=b$; but this means that both of them cannot be in the same time bigger (or smaller) than $\frac{-b}{2a}$; which is a contradiction since $f'(\frac{-b}{2a})=0$ and $\frac{-b}{2a}$ is an extremum point of $f$. Hence:


$$ f(x)=f(y), x\neq y\iff a(x+y)=b$$

Therefore it is enough to find the function for $x>\frac{-b}{2a}$. 


*

*See that after using the information derived above:
$$
f(ax^2+bx+\frac{b^2-2b}{4a})=af(x)^2+bf(x)+\frac{b^2-2b}{4a}\implies\\
f\left(a(x+\frac{b}{2a})^2-\frac{b}{2a}\right)=a\left(f(x)+\frac{b}{2a}\right)^2-\frac{b}{2a}.
$$.
Note that if $a>0$, then $x>-\frac{b}{2a}\implies f(x)>-\frac{b}{2a}$; w.l.o.g. we assume $a>0$ and $x>-\frac b{2a}$. 
$$
af\left(a(x+\frac{b}{2a})^2-\frac{b}{2a}\right)+\frac{b}{2}=\left(af(x)+\frac{b}{2}\right)^2
$$
See that defining $g(x)=af(x)+\frac b2$, we get:
$$
g(a(x+\frac{b}{2a})^2-\frac{b}{2a})=g(x)^2
$$
Define the function:
$$
h(x)=\frac{\log g(x)}{\log(ax+\frac b2)}.
$$
Then we get:
$$
h(a(x+\frac{b}{2a})^2-\frac{b}{2a})=h(x)
$$
This function can be shown to be a constant by constructing  a decreasing sequence $u_n$  from an arbitrary $u$ and showing that $h(u)$ is equal to the limit of $u_n$ which is a constant value independent of $u$. Hence:
$$
\frac{\log g(x)}{\log(ax+\frac b2)}=C\implies g(x)=(ax+\frac b2)^C\\
\implies f(x)=\frac 1a\left((ax+\frac b2)^C-\frac b2\right).
$$ 
By plugging in to the original question, it turns out that $C=\sqrt 2$. Extending it for all $x$'s, we get:



$$
f(x)=\frac 1a\left(|ax+\frac b2|^{\sqrt 2}-\frac b2\right) \text{ if } b^2-4ac=2b.
$$

A: This is only a partial answer, but it might be of some help.
If $f(x)=mx+d$, then $f(f(x))=m(mx+d)+d=m^2x+(m+1)d$, so any triple of the form $(0,b,c)$ with $b\ge0$ works, by taking $m=\sqrt b$ and $d=c/(1+\sqrt b)$.
Likewise, if $f(x)=m|x|^\sqrt2$, then $f(f(x))=m|(m|x|^\sqrt2)|^\sqrt2=m|m|^\sqrt2x^2$, so any triple of the form $(a,0,0)$ works, by taking $m=sgn(a)|a|^{1/(1+\sqrt2)}$.
So it looks to me like there are two natural questions:  1) Does $a\not=0$ force $b=c=0$? and 2) are there any triples with $b\lt0$?
A: I will divide the Case to these $3$:
$$
\textbf{Case 1. }\bf{(b-1)^2-4ac>0.} \\
\textbf{Case 2. }\bf{(b-1)^2-4ac=0.} \\
\textbf{Case 3. }\bf{(b-1)^2-4ac<0.}
$$
This is my solution:
\begin{align}
& \textbf{Case 1. } \bf{(b-1)^2-4ac > 0:} \\
\ \\
&\text{Let's think about $t_2$ which satisfies } f^2(t_2)=t_2. \\
\Rightarrow \ t_2 =  \ & a{t_2}^2+bt_2+c. \Rightarrow a{t_2}^2+(b-1)t_2+c=0. \\
\ \\
\therefore \ t_2 =  \ & \frac{-b+1\pm\sqrt{b^2-2b+1+4ac}}{2a}. \\
\Rightarrow \ & \text{I will let $t_2$=$\alpha_1$ and $\beta_1$.}
\ \\
&\text{Now, let's think about $t_4$ which satisfies $f^4(t_4)=t_4$.} \\
\Rightarrow \ t_4 \ & = a(a{t_4}^2+bt_4+c)^2+b(a{t_4}^2+bt_4+c)+c \\ 
&=a^3{t_4}^4 + 2a^2b{t_4}^3+(2a^2c+ab^2+ab){t_4}^2+(2abc+b^2){t_4}+bc+c. \\
\therefore \ t_4 \ & \text{has $4$ solutions, including $t_2$.} \\
\ \\
&\text{I will let $t_4=\alpha_1, \beta_1, \alpha_2, \beta_2$.} \\
\ \\
\Rightarrow &\text{I'll define $\alpha_1$chain, $\beta_1$chain, $\alpha_2$chain and $\beta_2$chain:} \\
\ \\
\alpha_1\text{chain}: & \begin{bmatrix} \alpha_1 & \rightarrow & f(\alpha_1) \\ \uparrow & 
\fbox{$2_\text{cyc}$} & \downarrow \\ f(\alpha_1) &\leftarrow &  \alpha_1 \end{bmatrix} \\
\ \\
\beta_1\text{chain}: & \begin{bmatrix} \beta_1 & \rightarrow & f(\beta_1) \\ \uparrow & 
\fbox{$2_\text{cyc}$} & \downarrow \\ f(\beta_1) &\leftarrow &  \beta_1 \end{bmatrix} \\
\ \\
\alpha_2\text{chain}: & \begin{bmatrix} \alpha_2 & \rightarrow & f(\alpha_2) \\ \uparrow & 
\fbox{$4_\text{cyc}$} & \downarrow \\ f^3(\alpha_2) &\leftarrow &  f^2(\alpha_2) \end{bmatrix} \\
\ \\
\beta_2\text{chain}: & \begin{bmatrix} \beta_2 & \rightarrow & f(\beta_2) \\ \uparrow & 
\fbox{$4_\text{cyc}$} & \downarrow \\ f^3(\beta_2) &\leftarrow &  f^2(\beta_2) \end{bmatrix} \\
\ \\
&\text{I will let set } T_4=\{t_4\}=\{\alpha_1, \beta_1, \alpha_2, \beta_2\}. \\
\Rightarrow \ &f^n(\alpha_1), f^n(\alpha_2), f^n(\beta_1), f^n(\beta_2) \in T_4. \\
\ \\
&\text{Looking at $\alpha_2$ and $\beta_2$, you can easily show the contradiction.} \\
&\text{(just try to assume the value of $f(\alpha_2)$, for example.)} \\
\ \\
\therefore & \not\exists f: \mathbb{R} \to \mathbb{R} \text{ s.t. } f(f(x))=ax^2+bx+c, (b-1)^2-4ac>0.
\end{align}
For Case 2, you can just think about 3 variables with $t_2$ and $t_4$.($n(T_4)=3, \ \exists! \ t_2.$)
For Case 3, there is a solution for few $f$.
I will let you think about Case 2. I can't answer about Case $3$, because this answer just wanted to show that $f$ satisfying the OP doesn't exist, totally.
