How to find $f$ and $g$ if $f\circ g$ and $g\circ f$ are given? The question is: Let $f:\mathbb R\rightarrow \mathbb R$ and $g:\mathbb R\rightarrow \mathbb R$ be two functions such that $(f\circ g)(x)=4x^2+4x+1$ and $(g\circ f)(X)=x^2+2x+2$. Find $f(x)$ and $g(x)$.
I solved the problem like this.
Because both the compositions are polynomials is $x$, it is clear that $f(x)$ and $g(x)$ also both will be polynomials in $x$. Let $\deg (f)=m, \deg (g)=n$. Then it follows that $m, n\leq 2$ cause otherwise the compositions will be polynomials of degree more than 2.
Now if both $m, n=2$ then $f\circ g, g\circ f$ will be of degree more than 2, contradiction again. Hence one of them will be 2 and the other will be 1. WOLOG let us assume that $m=2, n=1$.
Then we assume $f(x)=ax^2+bx+c, g(x)=dx+e$ we have $f\circ g=a(dx+e)^2+b(dx+e)+c= ad^2x^2+(2ade+bd)x+ae^2+be+c$ and $g\circ f=d(ax^2+bx+c)+e=dax^2+dbx+dc+e$. Equating the coeffcicients of similar terms, we then get the values of $a, b, c, d, e$ and hence the problem is solved.
Please inform me whether my approach to solve this problem was correct or not.
 A: If it is given that $f$ and $g$ are polynomials then the product of their degrees must be $2$ (the degree of the composition of two polynomials is the product of their individual degrees), therefore one of the two must be quadratic and the other linear. I do not think that you can assume $\deg f=2$ without loss of generality.
If $\deg f=2$ and $\deg g=1$ then $g$ must be increasing, i.e., its leading coefficient is positive. The graph of $f,$ like the graph of $f\circ g,$ is tangent to the $X$ axis (discriminant $0$) and $f\circ g$ reaches its minimum $0$ when $x=-\frac12.$ The minimum of $g\circ f$ is $g(f(-1))=1$ and, because $g$ is strictly increasing, is reached when $f(x)$ reaches its minimum $0.$ Therefore the $Y$-intercept of $g$ is 1. This also tells us that $f$ reaches its minimum at $x=-1$ and we already know that $g(-\frac12)=-1$. Therefore
$$g(x)=4x+1;\ f(x)=a(x+1)^2$$
Plugging this into $g(f(x))=x^2+2x+2$ gives $a=\frac14,$ and then into $f(g(x))=4x^2+4x+1$ verifies this.
Now suppose $\deg g=2$ and $\deg f=1.$ Then $f$ must increasing (positive leading coefficient). The graph of $g$, like the graph of $g\circ f,$ has minimum $1.$ The function $g\circ f$ reaches this minimum when $x=-1.$ The minimum of $f\circ g$ is $f(g(-\frac12))=0$ and, because $f$ is strictly increasing, is reached when $g(x)$ reaches its minimum $1.$ Therefore $f(1)=0.$ This also tells us that $g$ reaches its minimum at $x=-\frac12$ and we already know that $f(-1)=-\frac12.$ Therefore
$$f(x)=\frac14x-\frac14;\ g(x)=a(2x+1)^2+1$$
Plugging this into $f(g(x))=4x^2+4x+1$ gives $a=4,$ and then into $g(f(x))=x^2+2x+2$ verifies the result.
Without any restriction on $f$ and $g$ the solutions can be very wild. If $(f_1,g_1)$ and $(f_2,g_2)$ are distinct solutions where all 4 functions leave a proper subset $A\subset\mathbb R$ and its complement intact, then two new solutions can be obtained by applying one solution to the numbers in $A$ and the other to its complement. As an example, the two polynomial solutions above map algebraic numbers to algebraic numbers and transcendental numbers to transcendental numbers.
A: If you disregard the requirement that $f(x)$ be real, there is a non-polynomial solution
$$ f(x) = \dfrac{\sqrt{x-1}}{4} -\dfrac{1}{2},\ g(x) = 1 + 4 (8 x^2 + 8 x + 3)^2$$
