Spivak's Calculus: Chapter 2 Question 8 I'm working through Spivak's Calculus (4th edition) and came across this question:

For what numbers $a$, $b$, $c$ and $d$ will the function
$$ f(x) = {ax+b \over cx + d} $$
satisfy $f(f(x)) = x$ for all $x$ (for which this function makes sense)?

It is easy enough to show that $ f(f(x)) = x $ implies 
$$ (ac + cd)x^2 + (d^2 - a^2)x - b(a+d) = 0 $$
so each coefficient in the polynomial needs to be zero:
\begin{align} 
& ac + cd = c(a + d) = 0, & \text{ so } a=-d \text{ or } c=0 \\
& d^2 -a^2 = 0, & \text{ so } a = -d \text{ or } a = d \\
& b(a+d) = 0, & \text{ so } a = -d \text{ or } b=0 
\end{align}
We can simplify the logic down to two cases: either $a = -d$ or ($a=d$ and $c=b=0$). We also need $c$ and $d$ to not both be zero, otherwise $f$ wouldn't make sense at all.
Now then, while playing around with this function on a graphing tool, I came across another restriction: if $a=-d$ and $c = -a^2$ and $b = 1$, then $f(x) = -{1 \over a}$. Clearly a constant function isn't its own inverse, but I can't for the life of me see how I could have possibly discovered this restriction without sheer dumb luck. Is there something in the symbols that I'm missing, or some other method to use in the future besides "think really really really hard about this specific function?"
 A: $f(f(x))=x$ means that precisely what you wrote (i'm just rewriting it)
\begin{align*}
\dfrac{a\left(\dfrac{ax+b}{cx+d}\right)+b}{c\left(\dfrac{ax+b}{cx+d}\right)+d}&=x\\
\dfrac{\left(\dfrac{a^2x+ab+bcx+bd}{cx+d}\right)}{\left(\dfrac{acx+bc+cdx+d^2}{cx+d}\right)}&=x\\
\dfrac{a^2x+ab+bcx+bd}{acx+bc+cdx+d^2}&=x\\
a^2x+ab+bcx+bd&=acx^2+bcx+cdx^2+d^2x\\
(ac+cd)x^2+(d^2-a^2)x-ab-bd&=0\\
c(a+d)x^2+(d+a)(d-a)x-b(a+d)&=0
\end{align*}
Note that you can factor $(a+d)$ out of this condition:
$$(a+d)\left[cx^2+(d-a)x-b\right]=0\tag{$*$}$$
and then you don't need to deal with three equations at the same time: a product is zero iff one of the terms is zero! That is, $f(f(x))=x$ for all $x$ iff equation $(*)$ is valid for all $x$, iff one of the following holds:


*

*$a=-d$; or

*$cx^2+(d-a)x-b=0$; which means that $a=d$ and $b=c=0$.


Remark: I am being a little loose with the order of quantifiers (namely, "for all $x$") here. You should see the equation $(*)$ as the following: the product of the polynomials $a+d$ and $cx^2+(a-d)x-b$ is zero. Then use the following: a product of two polynomials $p(x)$ and $q(x)$ is zero iff one of them is zero. To see this, just notice that at least one of $p(x)$ or $q(x)$ will have infinitely many roots, so it needs to be the zero polynomial.

EDIT So there is indeed a need to be more precise: We assume that $f(f(x))=x$ on all of its domain. So the question is: How large is this domain?
There are two possibilities: First, if $ad-bc=0$, then the function $f(x)$ is constant! Indeed, suppose $ad-bc=0$. There are a few possibilities:


*

*Suppose $d=0$. Then either $b=0$ or $c=0$. However, for $f(x)$ to be well-defined we need $cx+d\neq 0$, so $c\neq 0$. Then $b=0$ and $f(x)=a/c$.

*Now suppose $d\neq 0$, so $a=bc/d$. Then
$$f(x)=\dfrac{ax+b}{cx+d}=\dfrac{\dfrac{bc}{d}x+b}{cx+d}=\dfrac{b}{d}.$$
So if $ad-bc=0$, then $f(x)$ is constant, call this value $F$. In fact, in the two cases above you can verify that $cF+d=0$, so $f(x)$ is not defined at $x=F$. This means that the domain of $f\circ f$ is empty, and therefore $f(f(x))=x$ holds vacuously!
As for the other case, we assume  that $ad-bc\neq 0$, so the function $f$ is not constant. In fact, it is injective with left inverse $f^{-1}(x)=\dfrac{dx-b}{-cx+a}$ (check this). Thus the domain of $f\circ f$ will be at least the domain of $f$ minus one point (the preimage under $f$ of the point where $f$ is not defined, namely $f^{-1}(-d/c)$, if this is defined), and it will be in particular infinite. So equation $(*)$ holds for an infinitude of points and argument follows.
A: If $c\ne 0$ then $f(-d/c)$ is undefined. So $c=0$ and $f(x)=(ax+b)/d,$ which is defined only when $d\ne 0.$ So $d\ne 0.$ Then $x=f(f(x))=(af(x)+b)/d=(a^2x+ab)/d^2 $ for all $x.$ So $0\ne d=\pm a,$ and $ab=0$ with $a\ne 0$, so $b=0.$ And $\forall x\,(f(x)=x)$ or $\forall x\,(f(x)=-x).$
Q: Suppose $a,b$ are not both $0$ and $c,d$ are not both $0.$ Then for what $a,b,c,d$ can we have $f(f(x))=x$ for every $x$ such that $cx+d\ne 0 \ne cf(x)+d?$
