let $f(x)$ be rational non constant polynomial and $f\circ f(x)=3f(x)^4-1$ then find the $f(x)$ let $f(x)$ be rational non constant polynomial and $f\circ f(x)=3f(x)^4-1$
then find the $f(x)$ .
My Try :
$$f(f(x))=3f(x)^4-1$$
Let $f(x)=ax^n+g(x)$ so :
$$a(ax^n+g(x))^n+g(ax^n+g(x))=3(ax^n+g(x))^4-1$$
$$a^2x^{n^2}+h(x)+k(x)=3ax^{4n}+l(x)-1$$
$$3ax^{4n}-a^2x^{n^2}-1=h(x)+k(x)-l(x)$$
Now what ?
 A: Note that $3f(x)^4 - 1 = g\circ f(x)$, where $g(x) = 3x^4 - 1$. So $f = g$, at least, is a solution. 
Now, if $f$ is a solution, we need to show that $f(x) = g(x)$. But note that $f \circ f(x) = g\circ f(x)$ for all $x$, so $(g - f) \circ f(x) = 0$, so $g - f$ is zero on the image of $f$, which, since $f$ is non-constant, contains some interval $(a,b)$, hence $g - f$ is everywhere zero, so $f = g$ is the only solution. 
A: The highest degree terms on each side of what you have are:
$a_n^{n+1}x^{n^2}  = 3a_n^4x^{4n}$
$n^2 = 4n\\ n = 4\\ a_4^5 = 3a_4^4\\
a_4 = 3$
But, there is a more direct approach.
$f(f(x)) = 3(f(x))^4 - 1$
Replace the $f(x)$ with $u$ everywhere we see it. 
$f(u) = 3u^4 - 1$
A: Note that, because I think that you supposed deg $g(x)\leq n-1$, from
$$
a_n(a_n x^n+g(x))^n+g(a_n x^n+g(x))=
3(a_n x^n+g(x))^4-1$$
the degree of the LHS is $n^2$, while on the other hand the degree of the RHS is $4n$. This is an equality between two polynomials, hence their degree must be the same. Then $n=4$. Moreover the two polynomials must have the same leading coefficient. The coefficient of $x^{16}$ in the LHS is $a_4^{5}$, while the coefficient in the RHS is $3a_4^{4}$, hence $a_4=3$.
Now I think that you have to put $f(x)=3x^4+\sum_{i=0}^{3} a_i x^i $ in the equation and compute
