Given $f(x) = \frac1{ax+b}$, for which $a$, $b$ such that $x_1=f(x_3) $, $ x_2=f(x_1) $, $x_3=f(x_2) $ are distinctive For real numbers $a$ and $b$ define
$$f(x) = \frac1{ax+b}$$
For which $a$ and $b$ are there three distinct real numbers $x_1$, $x_2$, $x_3$ such that $f(x_1) = x_2$, $f(x_2) = x_3$ and $f(x_3) = x_1$?
I tried to isolate a/b, and I found 3 values: 
f(x1) = x2 ---> 1/(a.x1 + b) = x2 ---> a.x1.x2 + b.x2 = 1 ---> I f(x2) = x3 ---> 1/(a.x2 + b) = x3 ---> a.x2.x3 + b.x3 = 1 ---> II f(x3) = x1 ---> 1/(a.x3 + b) = x1 ---> a.x1.x3 + b.x1 = 1 ---> III make I=II then, a/b = (x3 - x2)/x2.(x1 - x3) ---> x1 ≠ x3, for example, but i don't know how to proceed from here.
 A: Assume $a\ne 0$. Then, $$x_1 = \frac1{ax_3+b}= \frac1{a\frac1{ax_2+b}+b}=\frac1{a\frac1{a\frac1{ax_1+b}+b}+b}$$
which leads to
$$(a+b^2)(ax_1^2+bx_1-1) = 0\tag 1$$
and similarly, 
$$(a+b^2)(ax_2^2+bx_2-1) = 0,\>\>\>\>\>(a+b^2)(ax_3^2+bx_3-1) = 0\tag 2$$
If $a+b^2\ne 0$, then the solutions are
$$x_1 = x_2=x_3=\frac{-b\pm\sqrt{b^2+4a}}{2a}$$
and there are no real values of $a$ and $b$ that would make the three $x$'s distinct. On the other hand, if $a+b^2=0$, there are infinite number of solutions to (1) and (2) which can all be distinct. Thus, the values of $a$ and $b$ satisfy 
$$a+b^2=0$$
A: Here is a linear algebra approach.  It is long because I am trying to explain what is going on.  However, the approach of pairing a map of the form $f(x)=\frac{ax+b}{cx+d}$ with the matrix $A=\begin{pmatrix}a&b\\c&d\end{pmatrix}$ is very useful.  For example, if $f(x)=\frac{ax+b}{cx+d}$ and $\tilde{f}(x)= \frac{\tilde{a}x+\tilde{b}}{\tilde{c}x+\tilde{d}}$ correspond to the matrices $A=\begin{pmatrix}a&b\\c&d\end{pmatrix}$ and $\tilde{A}=\begin{pmatrix}\tilde{a}&\tilde{b}\\\tilde{c}&\tilde{d}\end{pmatrix}$, respectively, then $f\circ \tilde{f}$ corresponds to the matrix $A\tilde{A}$.
Back to the question, recall that $f(x)=\frac{1}{ax+b}=\frac{0x+1}{ax+b}$.  First of all $a\ne 0$ (otherwise $f$ is constant which obviously doesn't fit the bill).  Let $A$ be the matrix $\begin{pmatrix}0&1\\a&b\end{pmatrix}$.  The fact that $a\ne 0$ means that $A$ has rank $2$.  Therefore $A$ is invertible (whence the eigenvalues of $A$ are non-zero).  
Associate a real number $x$ with any non-zero vector $\xi=\begin{pmatrix}u\\v\end{pmatrix}$ s.t. $v\ne 0$ and $\frac{u}{v}=x$.  We use the notation $x\sim\xi$ in this case.   Also associate $\infty$ with vectors $\zeta=\begin{pmatrix}w\\0\end{pmatrix}$ with $w\ne 0$.  (Using $\infty$ is useful here.  We can for example set $f(-b/a)=\infty$ and $f(\infty)=0$.)   Observe that if real numbers $x$ and $y$ satisfy $x\sim\xi$ and $y\sim \xi$
for some non-zero vector $\xi$, then $x=y$.
Suppose that $x\sim \xi$.  Then show that $f(x) \sim A\xi$.  Now let $x_1$, $x_2$, and $x_3$ be distinct real numbers s.t. $f(x_1)=x_2$, $f(x_2)=x_3$, and $f(x_3)=x_1$.  If $x_i\sim \xi_i$ for $i=1,2,3$, then $$x_2=f(x_1)\sim A\xi_1.$$  But we have $x_2\sim \xi_2$ also.  By the definition of $\sim$, this means $A\xi_1=r\xi_2$ for some non-zero real number $r$.  Now
$$x_3=f(x_2)\sim A\xi_2$$
and $x_3\sim \xi_3$.  Using the same argument, $A\xi_2=s\xi_3$ for some non-zero real number $s$.  Finally from
$$x_1=f(x_3)\sim A\xi_3$$
and $x_1\sim \xi_1$, we get that $A\xi_3\sim t\xi_1$ for some non-zero real number $t$.
Therefore
$$A^3\xi_1=A^2(A\xi_1)=A^2(r\xi_2)=rA^2\xi_2=rA(A\xi_2)=rA(s\xi_3)=rs(A\xi_3)=rst\xi_1.$$
Therefore $A^3$ has a (non-zero) real eigenvalue $rst$.  Since $A$ is a real $2\times 2$ matrix, either it has two real eigenvalues or two non-real conjugate eigenvalues.  We want to prove that $A$ doesn't have a real eigenvalue.
Suppose for the sake of contradiction that both eigenvalues of $A$ are real numbers $\lambda_1,\lambda_2$. Note that
$$\det(A^2+kA+k^2I)=(\lambda_1^2+k\lambda_1+k^2)(\lambda_2^2+k\lambda_2+k^2)>0$$
if $k=\sqrt[3]{rst}$.  Hence $(A^2+aA+a^2I)$ is invertible.
Now $$(A^2+kA+k^2I)(A-kI)\xi_1=(A^3-k^3I)\xi_1=A^3\xi_1-k^3\xi_1=A^3\xi_1-rst\xi_1=0.$$
So $$A\xi_1-k\xi_1=(A-kI)\xi_1=(A^2+kA+k^2I)^{-1}0=0.$$  That is $A\xi_1=k\xi_1$.  But we know $A\xi_1=r\xi_2$.  Therefore $k\xi_1=r\xi_2$.  That means $x_1$ and $x_2$ are associated to $\xi_2$.  This shows that $x_1=x_2$, contradicting the assumption that $x_1$, $x_2$, and $x_3$ are distinct.  Hence the eigenvalues of $A$ must be non-real complex numbers $\lambda$ and $\bar{\lambda}$.
Now $rst$ is an eigenvalue of $A^3$, so we must have $\lambda^3=k^3$ or $\bar{\lambda}^3=k^3$ (recalling that $k^3=rst$ is real).  In the latter case, $\lambda^3=\overline{\left(\bar{\lambda}^3\right)}=\overline{k^3}=k^3$, so we still have $\lambda^3=k^3$ anyhow.  Therefore, $\lambda$ is an eigenvalue of $A$ which is non-real but $\lambda^3$ is the real number $k^3$.  Since $\lambda$ is non-real, $\lambda\ne k$, so
$$\lambda^2+k\lambda+k^2=\frac{\lambda^3-k^3}{\lambda-k}=0.$$
This shows that the characteristic polynomial of $A$ is
$$\det(A-qI)=q^2+kq+k^2.$$
But $$\det(A-qI)=\det\begin{pmatrix}-q&1\\a&b-q\end{pmatrix}=q^2-bq-a.$$
Therefore $-b=k$ and $-a=k^2$, so
$$a=-k^2=-(-k)^2=-b^2.$$
And we recall that $a\ne 0$, so the necessary condition is $a=-b^2\ne 0$.  This condition is also sufficient because when $a=-b^2\ne0$,
$$A^3=-b^3I.$$
This means that $f\circ f\circ f$ is the identity.  Since $A$ has no real eigenvalue, $f$ has no fixed point.  Consequently, for any $x\in \Bbb R\cup\{\infty\}$, the values of $x$, $f(x)$, and $f\circ f(x)$ are distinct, and $f\circ f\circ f(x)=x$.
