$\newcommand{\Cpx}{\mathbf{C}}$You're secretly describing certain automorphisms (holomorphic bijections with holomorphic inverse) of the open unit disk.
The open unit disk plays a starring role in complex analysis: It turns out that every connected, simply-connected, proper open subset of the complex plane is "biholomorphic" to the open disk (the so-called Riemann mapping theorem).
Edit: There was a mistake in my original answer, in that part (b) doesn't describe all automorphisms of the disk, but only a certain family of involutions (automorphisms that are their own inverse).
To fill in some of the background, here's a brief sketch of the Riemann sphere as the complex projective line, Möbius transformations as $2 \times 2$ complex matrices, and a more detailed interpretation of part (b).
The complex projective line is the set of (complex) lines through the origin in $\Cpx^{2}$. Algebraically, consider ordered pairs $(z_{0}, z_{1})$ of complex numbers (not both zero) modulo the equivalence relation
$$
(z_{0}, z_{1}) \sim (\lambda z_{0}, \lambda z_{1})\quad\text{for all $\lambda \neq 0$.}
$$
Associating $z$ in $\Cpx$ with the line through $(z, 1)$ (labeled $V_{0}$ in the figure below), we represent every line except the line through $(1, 0)$; since $(z, 1) \sim (1, \frac{1}{z})$ for $z \neq 0$, it's reasonable to view $(1, 0)$ as "$\infty$"; doing this identifies the complex projective line with the Riemann sphere. (There's a bit more detail in another post, from which the following figure is recycled.)
An "automorphism" of the projective line/Riemann sphere is a bijection that is meromorphic in the local coordinate $z$. To make a moderately lengthy story short, every automorphism of the projective line turns out to be induced by a linear automorphism of $\Cpx^{2}$, via the correspondence
$$
\left[\begin{array}{@{}rr@{}}
a & b \\
c & d \\
\end{array}\right]
\left[\begin{array}{@{}c@{}}z \\ 1\end{array}\right]
= \left[\begin{array}{@{}c@{}} az + b \\ cz + d\end{array}\right]
\sim \left[\begin{array}{@{}c@{}}\dfrac{az + b}{cz + d} \\ 1\end{array}\right],\quad ad - bc \neq 0.
$$
(This is the origin of the term "fractional linear" transformation.) In particular, composition of Möbius transformations corresponds to matrix multiplication.
The coefficient matrix can be multiplied by a non-zero constant without changing the associated Möbius transformation, and it's customary to normalize by taking $ad - bc = 1$, i.e., to identify projective automorphisms with $SL(2, \Cpx)/{\pm}$.
As to part (b) of your question: The Möbius transformation $f(z) = \dfrac{z - a}{\overline{a}z - 1}$ corresponds to the matrix
$$
A = \left[\begin{array}{@{}rr@{}}
1 & -a \\
\overline{a} & -1
\end{array}\right].
$$
Part (b) asserts that if $|a| < 1$, then $|z| < 1$ if and only if $|f(z)| < 1$, and $|z| = 1$ if and only if $|f(z)| = 1$. In other words, $f$ maps the open unit disk to itself, and maps the unit circle to the unit circle. Moreover, $A^{2} = (1 - |a|^{2}) I \sim I$, so $f$ is an involution of the disk.
Incidentally, the mapping $g(z) = -1/z$ in part (a) is an involution of the upper half plane. It's not difficult to show that the Möbius transformations
$$
\phi(z) = \frac{z - i}{z + i}
\leftrightarrow \left[\begin{array}{@{}rr@{}}
1 & -i \\
1 & i \\
\end{array}\right],\quad
\phi^{-1}(z) = i\frac{1 + z}{1 - z}
\leftrightarrow \left[\begin{array}{@{}rr@{}}
i & i \\
-1 & 1 \\
\end{array}\right]
$$
map the open upper half plane to the open disk (and vice versa), and it's an easy exercise to check that $(\phi \circ g \circ \phi^{-1})(z) = -z$, the "$a = 0$" mapping in part (b).
