functional equation $f(f(x))=af(x)+bx$ Let $0<a,b<\frac{1}{2}$.
Find all continuous functions $f$ satisfying $f(f(x))=af(x)+bx$.

I proved $f$ is monotonic, but stucked at this point. 
How to use the bounds for $a,b$, why is it important to have $a,b\in(0,\frac{1}{2})$? 
 A: Assume $f(x_1)=f(x_2)$. Then 
$$ bx_1=f(f(x_1))-af(x_1)=f(f(x_2))-af(x_2)=bx_2$$
and hence (as $b\ne 0$) $x_1=x_2$. We conclude that $f$ is injective. As $f$ is also continuous, $f$ is strictly monotonous. Thereefore $f\circ f$ is strictly increasing.
Assume $L:=\lim_{x\to +\infty}f(x)$ exists. Then $f(L)=\lim_{x\to+\infty}(af(x)+bx)=\infty$, contradiction. Similarly, $\lim_{x\to -\infty}f(x)$ cannot exist. 
We conclude that $f$ is a continuous bijection $\Bbb R\to\Bbb R$.
Pick $x_0\in \Bbb R$ and define recursively $x_{n+1}=f(x_n)$. Then $x_{n+2}=ax_{n+1}+bx_n$ for all $n$. It is well-known that the solutiosn of this recursion are of the form
$$x_n=\alpha \lambda_1^n+\beta\lambda_2^n, $$
where $\lambda_{1,2}=\frac{a\pm\sqrt{a^2+4b}}{2}$ are the solutions of $X^2-aX-b=0$ and $\alpha,\beta$ are found by solving $x_0=\alpha+\beta$, $x_1=\alpha\lambda_1+\beta\lambda_2$. From the given restrictions for $a,b$, we find $-\lambda_1<\lambda_2<0<\lambda_1<1$. This implies that $x_n\to 0$, independent of $x_0\in\Bbb R$.
By continuity, $f(0)=0$.
Assume $x_n>0$ and $x_{n+2}\ge x_n$. As $f\circ f$ is strictly increasing, this implies that the subsequence $x_{n+2k}$ is non-decreasing, contradicting convergence to $0$. Therefore $x_n>0$ implies $x_{n+2}<x_n$.
Similarly, $x_n<0$ implies $x_{n+2}>x_n$.
As $f$ is bijective, we can extend the sequence $x_n$ no $n\in\Bbb Z$.


*

*If $\alpha\ne 0$, the $\lambda_1^n$ terms is dominant for $n\gg 0$. In particular, all $x_n$ with $n\gg 0$ have the same sign. It follows that the sequences $x_{2n}$ and $x_{2n+1}$ are both decreasing or both increasing. 

*If $\beta\ne 0$, the$\lambda_2^n$ term is dominant for $n\ll 0$. In particular, $x_n$ flips signs as $n\ll 0$. It follows that the sequences $x_{2n}$ and $x_{2n+1}$ have opposite monotonicity.
Both these observations together imply that  $\alpha$ and $\beta$ cannot both be non-zero.
Therefore, either $f(x)=\lambda_1x$ or $f(x)=\lambda_2 x$. By continuity of $f$, a switch between these two conditions can at most occur at $x=0$, but then $f$ would not be monotonous. We conclude that no switch occurs, so that the only possible solutions are
$$ f(x)=\lambda_1x\text{ for all }x\in\Bbb R$$
and 
$$ f(x)=\lambda_2x\text{ for all }x\in\Bbb R.$$
A: Consider a point $\tilde{x}$ satisfying $f(\tilde{x})=c\tilde{x}$. We note that this can be true of any point $\tilde{x}$ for arbitrary $c$, we find
\begin{equation}
f(f(\tilde{x}))=f(c\tilde{x})=af(\tilde{x})+b\tilde{x}=(ac+b)\tilde{x}=(a+bc^{-1})(c\tilde{x}),
\end{equation}
which gives an iterative relation between c and $\tilde{c}$ for the next iterate.
Define the iterative function $\tilde{f}(c)=a+bc^{-1}$.  Upon iterating this function many times, one approaches one of the fixed points, which are solutions to the quadratic $c^2-ac-b=0$, which are $\frac{a\pm\sqrt{a^2+4b}}{2}$. Therefore, upon iterating arbitrarily many times, one approaches a point for which $f(\tilde{x})\approx c^*\tilde{x}$, where $c^*$ is one of the solutions of the quadratic.  
The iterative equation $\tilde{f}$ has an inverse given by $\tilde{f}^{-1}(y)=\frac{b}{y-a}$, with the same fixed points.  Therefore, iterating backwards or forwards from any point, one always eventually gets to a point where $f(\tilde{x})\approx c^*\tilde{x}$.  Therefore, iterating either forward or backward, one can always get to a contraction mapping that approaches $x=0$.  Choosing $x$ infinitesimally small, one can iterate back out, and of course it also must approach a similar $c^*$.  Therefore the function should be homogeneous of the form $f(x)=c^*x$ for the solutions of the quadratic.
I don't know if this is completely rigorous but I find it convincing at least.
