Find the function such $f(x+f(y)+xf(y))\ge y+f(x)+yf(x),\forall x,y\in(-1,+\infty)$ Let $ f ( x ) : ( - 1 , + \infty ) \to ( - 1 , + \infty ) $ be a continuous monotonic function, such that $ f ( 0 ) = 0 $, and
$$ f ( x + f ( y ) + x f ( y ) ) \ge y + f ( x ) + y f ( x ) \quad \forall x , y \in ( - 1 , + \infty ) \text {.} $$
Find $ f ( x ) $.
Let $ x = 0 $, then we have
$$ f ( f ( y ) ) \ge y \text {.} $$
 A: Throughout this answer, increasing functions (resp. decreasing functions) are the same as non-decreasing functions (resp. non-increasing functions).
1. Short answer
As pointed out by other users, if we put $\tilde{f}(x) = f(x-1) + 1$ then the $g : (0,\infty) \to (0,\infty)$ is a continuous monotone function satisfying
$$ \tilde{f}(1) = 1 \qquad \text{and} \qquad \forall x, y > 0 \ : \ \tilde{f}(x)y \leq \tilde{f}(x\tilde{f}(y)). \tag{1}$$
Except for $\tilde{f}(x) = x$ or $\tilde{f}(x) = x^{-1}$ which even solves the equality-version of $\text{(1)}$, there are non-trivial solutions of this functional inequality. Here is one simple example:

Example 1. For each $a \geq 1$, the function $\tilde{f} : (0,\infty) \to (0,\infty)$ given by
$$ \tilde{f}(x) = \begin{cases} x^{1/a}, & 0 < x < 1 \\ x^a & x \geq 1 \end{cases} $$
solves $\text{(1)}$. This may be proved by a tedious case division, but it is an easy consequence of what we develop in the sequel.


2. Longer answer
If we set $g(x) = \log \tilde{f}(e^x)$, then $g$ is a continuous monotone function on $\mathbb{R}$ and $\text{(1)}$ is equivalent to
$$ g(0) = 0 \qquad \text{and} \qquad \forall x, y \in \mathbb{R} \ : \ g(x) + y \leq g(x + g(y)). \tag{2} $$
This is our starting point of investigation. (Note: The role of $g$ is simply to replace multiplication by addition, and you can equally develop a multiplicative analogue of what follows.)

Lemma 1. If $g$ solves $\text{(2)}$, the same is true for $h(x) = -g(x)$.



*

*Proof. This immediately follows by rearranging the inequality $g(x-g(y)) + y \leq g(x)$, which follows from $\text{(2)}$ by replacing $x$ by $x - g(y)$. ////



Lemma 2. $g$ is surjective.



*

*Proof. Since $g$ is continuous, $g(\mathbb{R})$ is an interval. Plugging $x = 0$ to $\text{(2)}$, we have $y \leq g(g(y))$ and hence $g(\mathbb{R})$ is unbounded above. Applying the same argument to $-g$, it is also unbounded below. Therefore $g(\mathbb{R}) = \mathbb{R}$. ////



Lemma 3. $g(a) = 0$ implies $a = 0$.



*

*Proof. By Lemma 1, we may assume without losing the generality that $g$ is increasing. We first prove that $g(a) \leq 0$ implies $a \leq 0$. Indeed, if $g(a) \leq 0$, then by $\text{(2)}$ we have
$$ a = g(0) + a \leq g(0 + g(a)) \leq g(0) = 0. $$
Next, assume that $g(a) = 0$ but $a \neq 0$. By our previous observation, we must have $a < 0$. Since $g$ is surjective, we can pick $y$ such that $g(y) = -a > 0$. Then by $\text{(2)}$,
$$ y = g(a) + y \leq g(a + g(y)) = g(0) = 0, $$
from which we obtain the contradiction $g(y) \leq 0$. ////

Lemma 4. If $g$ is increasing, then $h(x) = -g(x)$ satisfies $h(h(x)) = x$.



*

*Proof. Applying $\text{(2)}$ twice, we have
$$ 0
= g(x) - g(x)
\leq g(x + g(-g(x)))
\leq g(g(-g(x) + g(x)))
= 0. $$
So $g(x + g(-g(x))) = 0$ and the conclusion follows from Lemma 3. ////
Now utilizing both Lemma 1 and Lemma 4, we immediately obtain the following proposition:

Proposition. Let $h : \mathbb{R} \to \mathbb{R}$ be continuous and monotone decreasing. Then the followings are equivalent:
  
  
*
  
*$h$ solves $\text{(2)}$.
  
*$g = -h$ solves $\text{(2)}$.
  
*$h(0) = 0$ and $h$ is a superadditive involution, i.e.,
  $$ \forall x \ : \ h(h(x)) = x \qquad \text{and} \qquad \forall x, y \ : \ h(x) + h(y) \leq h(x+y). $$

The role of this proposition is to simplify the condition $\text{(2)}$ to what is easier to study. 

Example 1 (continued). The function $\tilde{f}$ in Example 1 is constructed from the following choice of decreasing superadditive involution:
$$ h(x) = \begin{cases} -ax & x \geq 0 \\ -\frac{1}{a}x & x < 0 \end{cases}. $$
Indeed, it is immediate from definition that $h$ is continuous, decreasing and $h(h(x)) = x$. For the super-additivity, we directly compute that
$$ h(x+y) - h(x) - h(y) = \begin{cases} (a - a^{-1}) \min\{|x|,|y|\}, & xy < 0 \\ 0, & xy \geq 0. \end{cases}. $$

A: Let's look at a bit of a simpler question first: Is there a case in which equality holds? For $x = 0$, we would have that $f(f(y)) = y$. Thus, we must find a continuous, monotonic function on the interval $(-1, \infty)$ that is also involutive. Consider the following function:
\begin{equation}
f(x) = \begin{cases}
-1 + \frac{1}{x+1} & \text{if } x\in(-1,\infty)\\
0 & \text{else}
\end{cases}
\end{equation}
Clearly, $f(0) = 0$ as required, and it is easy to check that this function is involutive. Now, all one has to do is algebraically check that this function indeed satisfies the given equality. It turns out that both the RHS and LHS simplify to $\frac{y-x}{x+1}$.
Now, back to the original question: find $f(x)$ such that the $\textbf{inequality}$ is satisfied. Then, the involutivity condition is weakened to the inequality $f(f(y)) \geq y$. How can you modify the previous function? I suggest playing around with it a little bit and trying some specific cases before coming up with a general class of functions that will work!
