Find all real functions $f$ such that $f(xf(y))=(1-y)f(xy)+x^2y^2f(y)$

Find all functions $f:\mathbb{R}\rightarrow \mathbb{R}$ such that for all real numbers $x,y$ we have: $f(xf(y))=(1-y)f(xy)+x^2y^2f(y)$

I have solved this problem but the solution is "bruteforce", so I wanted to ask if there is a more elegant way of approaching this.

• you have found all solutions by bruteforce? – Börge Oct 8 '15 at 11:18

I claim that the only possible solutions to the functional equation $$f\big(xf(y)\big)=(1-y)f(xy)+x^2y^2f(y)\tag0\label0$$ are $$f(x)=0$$ and $$f(x)=x-x^2$$.
To show that, first let $$y=1$$ in \eqref{0} and you'll get $$f\big(xf(1)\big)=x^2f(1)$$. Now assuming $$f(1)\neq0$$ we'll have $$f(x)=\frac{x^2}{f(1)}$$ for every real number $$x$$, whcih leads to a contradiction using \eqref{0}. So we have: $$f(1)=0\tag1\label1$$ Next, let $$x=1$$ in \eqref{0} and you'll get: $$f\big(f(y)\big)=\big(1-y+y^2\big)f(y)\tag2\label2$$ Now, substituting $$f(y)$$ for $$y$$ in \eqref{0} and using \eqref{2} we get: $$f\Big(xf\big(f(y)\big)\Big)=\big(1-f(y)\big)f\big(xf(y)\big)+x^2f(y)^2f\big(f(y)\big)\\ \therefore f\Big(x\big(1-y+y^2\big)f(y)\Big)=\big(1-f(y)\big)f\big(xf(y)\big)+x^2f(y)^3\big(1-y+y^2\big)$$ So, if $$f(y)\neq0$$, letting $$x=\frac1{f(y)}$$ in the last equation and using \eqref{1}, we get: $$f\big(1-y+y^2\big)=\big(1-y+y^2\big)f(y)\ne0$$ Thus by \eqref{2}, $$f\big(f(y)\big)=f\big(1-y+y^2\big)\ne0$$ and so $$f\Big(f\big(f(y)\big)\Big)=f\Big(f\big(1-y+y^2\big)\Big)$$. Hence by \eqref{2}: $$\big(1-f(y)+f(y)^2\big)f\big(f(y)\big)=\Big(1-\big(1-y+y^2\big)+\big(1-y+y^2\big)^2\Big)f\big(1-y+y^2\big)\\ \therefore1-f(y)+f(y)^2=1-\big(1-y+y^2\big)+\big(1-y+y^2\big)^2\\ \therefore\Big(f(y)-\big(1-y+y^2\big)\Big)\Big(f(y)+\big(1-y+y^2\big)-1\Big)=0$$ $$\therefore f(y)=1-y+y^2\quad or\quad f(y)=y-y^2\tag3\label3$$ Now, if $$f(y)=1-y+y^2$$ then $$f\big(1-y+y^2\big)=\big(1-y+y^2\big)^2$$, but because $$1-y+y^2\neq0$$ we must have $$f\big(1-y+y^2\big)=1-\big(1-y+y^2\big)+\big(1-y+y^2\big)^2$$ or $$f\big(1-y+y^2\big)=\big(1-y+y^2\big)-\big(1-y+y^2\big)^2$$ by \eqref{3}. It's easy to check that the latter case leads to a contradiction and the former case leads to $$y=0$$ or $$y=1$$. But letting $$x=y=0$$ in \eqref{0} we have $$f(0)=0$$, and by \eqref{1} we have $$f(1)=0$$. So this case leads to a contradiction, too, since we assumed $$f(y)\neq0$$. So $$f(y)$$ can not be equal to $$1-y+y^2$$ and hence by \eqref{3}, $$f(y)=y-y^2$$.
Finally, if there is a real number $$y$$ such that $$y\neq0$$, $$y\neq1$$ and $$f(y)=0$$, then by \eqref{0} we get $$f(0)=(1-y)f(xy)+0$$ for every real number $$x$$, which means that $$f$$ is the constant zero function. This yields our first claim. It's easy to check that indeed the two mentioned functions satisfy \eqref{0}.
• If I may suggest a little simplification: from $(3)$ it easily follows that $f(x) = f(y) \implies$ $y = x$ or $y = 1-x$. Thus, when you get $f(f(y)) = f(1-y+y^2)$, you automatically get $f(y) = 1 - y+y^2$ or $f(y) = 1-(1- y +y^2)$, and can easily dismiss the first case because $1-y+y^2$ has no real roots, while $f$ does. – Ennar Oct 8 '15 at 14:16
• @Ennar I think that there's a problem with your argument. I think in that step, there's still possible that we have $f(y)=1-y+y^2$ for some $y$ and $f(y)=y-y^2$ for some other $y$. It may also be useful to note that, this part of the proof is about the real numbers for which we have $f(y)\neq0$ and so having roots is irrelevant. – Mohsen Shahriari Oct 8 '15 at 16:41