Let $f(x)$ be twice differentiable function, with $f(x) = x$ has $3$ roots Let $f(x)$ is twice differentiable increasing  function everywhere such that $f(x) = x$ has $3$ distinct root $\alpha ,\beta$ and $\gamma$ $(\alpha  < \beta  < \gamma )$ . Let $h(x) = \underset{n\to \infty }{\mathop{\lim }}\,\underset{n\,\,\,times}{\mathop{(f(f(...(f(x))))}}\,$.
(1)    If $f’’(x) > 0$ $\forall \,x\,\in \,(-\infty ,\beta )$,  $f”(x) < 0$ $ \forall x\,\in \,(\beta ,\infty ]\,$ and $f''(\beta )=0,\,$ then find $h(x)$
(2)    If $f(x)\ge x\,\forall \,x\,\in \,(-\infty ,\alpha ]\,\cup \,[\gamma ,\infty )$ and $f(x)\le x\,\forall \,x\,\in \,[\beta ,\gamma ]\,$ then find $h(x)$ 
I tried with a fucntion $k(x) = f(x)-x$ , such that $k(\beta)$ is $0$.
But how to do further.
 A: (1)
Let us first consider the case $x\in(\beta,\infty)$.
Let $g(x)=f(x)-x$. Note that $g''(x)<0$ on $x\in(\beta,\infty)$ and $g(\beta)=g(\gamma)=0$.
$\forall x\in(\beta,\gamma)$, $g(x)\ge 0$ since otherwise there exists $m\in[\beta,\gamma]$ s.t. $g(m)=\inf_{x\in[\beta,\gamma]}g(x)<0$, then $m\in(\beta,\gamma)$ and $g''(m)\le 0(\Rightarrow\Leftarrow)$
Moreover, $\forall x\in(\beta,\gamma),g(x)>0$ since otherwise $f(x)=x$.
Now, knowing that $g(x)\ge 0$ on $(\beta,\gamma)$ and $g(\gamma)=0$, we have $g'(\gamma)\le 0$. By integrating over $x$, we can obtain $\forall x>\gamma,g'(x)<0$ and $\forall x>\gamma, g(x)<0$.
Assume $\beta<x<\gamma$. $x<f(x)<f(\gamma)=\gamma$ since $g(x)>0$ on $(\beta,\gamma)$ and $f$ is increasing.
Assume $\gamma<x$. $\gamma=f(\gamma)<f(x)<x$ since $g(x)<0$ on $(\gamma,\infty)$ and $f$ is increasing.
-Here is what we've obtained up to now.

$x\in(\beta,\gamma)\Rightarrow f(x)\in(x,\gamma)$ and $x\in(\gamma,\infty)\Rightarrow f(x)\in(\gamma,x)$

By monotone convergence theorem, $\forall x\in(\beta,\gamma),\exists h(x)\in(\beta,\gamma]$. Here, $f(h(x))=h(x)\Rightarrow g(h(x))=0\Rightarrow h(x)=\gamma$. Similarly, $\forall x\in(\gamma,\infty),h(x)=\gamma$. Of course, $h(\gamma)=\gamma$.
Use the same argument to show $h(x)=\alpha$ for $x\in(-\infty),\beta)$.
$h(\beta)=\beta$ of course, so $h$ is 

$h(x)=\begin{cases}\alpha,x<\beta\\ \beta,x=\beta\\ \gamma,x>\beta\end{cases}$

(2)
Assume $h(x)$ exists on $x\in(\gamma,\infty)$. Note that $\forall x>\gamma, f(x)\ge x$. Thus, $h(x)\ge x>\gamma$. This contradicts $f(h(x))=h(x)$. Therefore, $h$ is undefined on $(\gamma,\infty)$.
Let us then think when $x\in(\beta,\gamma)$. Observe that in the first block(yellow) I only used $f(x)\in(x,\gamma)$ to show $h(x)=\gamma$. It works here as well.
Similarly, you can do that on $(-\infty,\alpha)$ to obtain $h(x)=\alpha$.
$x\in(\alpha,\beta)$ is more difficult. $g$ should be either always positive or always negative on $(\alpha,\beta)$ since otherwise we have $x\in(\alpha,\beta)$ s.t. $g(x)=0$ but that cannot happen.
When $g$ is positive, $h(x)=\beta$ for $x\in(\alpha,\beta)$ since $f(x)\in(x,\beta)$ on that interval.
When $g$ is negative, $h(x)=\alpha$ for the same reason.
Therefore, we have two possible answers for (2). (In fact, we have possible examples to both of them)

$h(x)=\begin{cases}\alpha,x\in(-\infty,\alpha]\\ \alpha\,or\,\beta,x\in(\alpha,\beta)\\ \beta,x=\beta\\ \gamma,x\in(\beta,\gamma]\\ \infty,x\in(\gamma,\infty)\end{cases}$

A: Let's look at the sequence defined by $x_{n+1} = f(x_n)$ where $f$ is increasing. Suppose $x_0<x_1$. Then it's straightforward by induction that $x_n<x_{n+1}$. Similarly, if $x_0>x_1$, then $x_n>x_{n+1}$. (The case $x_0=x_1$ is trivial.)  
Suppose now we have a stationary point $\xi$, meaning that $f(\xi)=\xi$. If $x_0<\xi$, then by induction (as $f$ is increasing): $x_n<\xi\implies x_{n+1}=f(x_n)<f(\xi)=\xi$. Hence $x_n<\xi$. Similarly, when $x_0>\xi$ we get $x_n>\xi$.
Suppose that $\lim x_n=L$ exists. Then as $f$ is continuous: $L=\lim x_{n+1}=\lim f(x_n)=f(\lim x_n)=f(L)$. So if $x_n$ is convergent, the limit must be one of the stationary points of $f$.  
Having established all this, notice that in your case $\alpha, \beta, \gamma$ are stationary points. The strategy in the following will be:


*

*identify the possible cases for $x_0$,

*given an $x_0$, find whether $x_1>x_0$ or $x_1<x_0$ to establish if $x_n$ increases or decreases,

*find the appropriate stationary point if such a point exists,

*conclude that $x_n$ either converges to that stationary point or diverges to $\pm\infty$,

*since the limit of $x_n$ is just $\lim f(f(...(f(x_0))))$, that will be $h(x)$.



$(1)$ First notice that $f'(\beta)>1$ - if not, then from $f''(x)<0$ for $x>\beta$, we would have $f'(x)<1$ for $x>\beta$, so $f(x)<x$ there and $\gamma$ couldn't exist. The qualitative picture of $f$ is then this: it comes in from $-\infty$ less steeply that $x$, intersects $x$ at $\alpha$, curves up to hit $x$ at $\beta$, curves back down to hit it at $\gamma$ and then goes on to $+\infty$ less steeply than $x$.
Since $f(x)>x$ on $(-\infty,\alpha)\cup(\beta,\gamma)$, we have that there $x_1=f(x_0)>x_0$, so $x_n$ is increasing. If we begin in the other two intervals, $(\alpha,\beta)\cup(\gamma,\infty)$, we analogously get $x_1<x_0$, so $x_n$ is decreasing. Since it's obvious which stationary points correspond to which case, we get:


*

*$x_0\in(-\infty,\beta): h(x_0)=\alpha$

*$x_0\in(\beta,\infty): h(x_0)=\gamma$

*$x_0=\beta: h(x_0)=\beta$

$(2)$ There's a bit of a problem here. I don't know if this was a typo, but your conditions leave unspecified whether $f(x)>x$ or $<x$ on $(\alpha,\beta)$. I'll just suppose $f(x)<x$ and the other option is very similar. Again, all you need to do is figure out where $f(x)<x$ and where $f(x)>x$ - in this case, $<$ is on $(\alpha,\beta)\cup(\beta,\gamma)$ and $>$ is on $(-\infty,\alpha)\cup(\gamma,\infty)$. Therefore:


*

*$x_0\in(-\infty,\beta): h(x_0)=\alpha$

*$x_0\in[\beta,\gamma): h(x_0)=\beta$

*$x_0=\gamma: h(x_0)=\gamma$

*$x_0\in(\gamma,\infty): h(x_0)=\infty$
