Find all real functions $f$ such that $f(xf(y))=(1-y)f(xy)+x^2y^2f(y)$

Find all functions $f:\mathbb{R}\rightarrow \mathbb{R}$ such that for all real numbers $x,y$ we have: $f(xf(y))=(1-y)f(xy)+x^2y^2f(y)$

I have solved this problem but the solution is "bruteforce", so I wanted to ask if there is a more elegant way of approaching this.

• you have found all solutions by bruteforce? – Börge Oct 8 '15 at 11:18

I claim that the only possible solutions to the functional equation $$f(xf(y))=(1-y)f(xy)+x^2y^2f(y)\qquad(1)$$ are $f(x)=0$ and $f(x)=x-x^2$.
To show that, first let $y=1$ in (1) and you'll get $f(xf(1))=x^2f(1)$. Now assuming $f(1)\neq0$ we'll have $f(x)=\frac{x^2}{f(1)}$ for every real number $x$, whcih leads to a contradiction using (1). So we have: $$f(1)=0\qquad(2)$$ Next, let $x=1$ in (1) and you'll get: $$f(f(y))=(1-y+y^2)f(y)\qquad(3)$$ Now, substituting $f(y)$ for $y$ in (1) and using (3) we get: $$f(xf(f(y)))=(1-f(y))f(xf(y))+x^2f(y)^2f(f(y))\\ \therefore f(x(1-y+y^2)f(y))=(1-f(y))f(xf(y))+x^2f(y)^3(1-y+y^2)$$ So, if $f(y)\neq0$, letting $x=\frac1{f(y)}$ in the last equation and using (2), we get: $$f(1-y+y^2)=(1-y+y^2)f(y)$$ Thus by (3), $f(f(y))=f(1-y+y^2)$ and so $f(f(f(y)))=f(f(1-y+y^2))$. Hence by (3): $$(1-f(y)+f(y)^2)f(f(y))=(1-(1-y+y^2)+(1-y+y^2)^2)f(1-y+y^2)\\ \therefore1-f(y)+f(y)^2=1-(1-y+y^2)+(1-y+y^2)^2\\ \therefore(f(y)-(1-y+y^2))(f(y)+(1-y+y^2)-1)=0\\ \therefore f(y)=1-y+y^2\quad or\quad f(y)=y-y^2\qquad(4)$$ Now, if $f(y)=1-y+y^2$ then $f(1-y+y^2)=(1-y+y^2)^2$, but because $1-y+y^2\neq0$ we must have $f(1-y+y^2)=1-(1-y+y^2)+(1-y+y^2)^2$ or $f(1-y+y^2)=(1-y+y^2)-(1-y+y^2)^2$ by (4). It's easy to check that the latter case leads to a contradiction and the former case leads to $y=0$ or $y=1$. But letting $x=y=0$ in (1) we have $f(0)=0$ and by (2) we have $f(1)=0$. So this case leads to a contradiction too, since we assumed $f(y)\neq0$. So $f(y)$ can not be equal to $1-y+y^2$ and hence by (4), $f(y)=y-y^2$.
Finally, if there is a real number $y$ such that $y\neq0$, $y\neq1$ and $f(y)=0$, then by (1) we get $f(0)=(1-y)f(xy)+0$ for every real number $x$, which means that $f$ is the constant zero function. This yields our first claim. It's easy to check that indeed the two mentioned functions satisfy (1).
• If I may suggest a little simplification: from $(3)$ it easily follows that $f(x) = f(y) \implies$ $y = x$ or $y = 1-x$. Thus, when you get $f(f(y)) = f(1-y+y^2)$, you automatically get $f(y) = 1 - y+y^2$ or $f(y) = 1-(1- y +y^2)$, and can easily dismiss the first case because $1-y+y^2$ has no real roots, while $f$ does. – Ennar Oct 8 '15 at 14:16
• @Ennar I think that there's a problem with your argument. I think in that step, there's still possible that we have $f(y)=1-y+y^2$ for some $y$ and $f(y)=y-y^2$ for some other $y$. It may also be useful to note that, this part of the proof is about the real numbers for which we have $f(y)\neq0$ and so having roots is irrelevant. – Mohsen Shahriari Oct 8 '15 at 16:41