For two circles $\alpha$ and $\beta$ with radical axis $l$, the pencil $\alpha\beta$ is the set of circles $\gamma$ which share this radical axis, i.e. the radical axis of $\gamma$ and $\alpha$ is $l$ as well (and thus automatically also the radical axis of $\gamma$ and $\beta$). In the picture below, the pencil is drawn in solid lines (black), and the common axis is the vertical (grey) line.
If and only if the circles do not intersect, there exists a circle (dashed red in the picture), which is centered at the intersection of $l$ and the common axis of symmetry $g$ of the pencil, and orthogonal to all circles of the pencil. This circle intersects the axis of symmetry in the two limiting points of the pencil. These can be thought to be the two circles of zero radius belonging to the pencil.
Inverting in a circle (dotted blue in the picture) centered at one of those points will leave the axis of symmetry invariant. The dashed circle, however, becomes a line orthogonal to $g$, since it passes through the center of inversion. A circle belonging to the pencil will be transformed to another circle orthogonal to both lines, which is thus necessarily centered at their intersection. Therefore, the pencil is transformed into a pencil of concentric circles, all centered at the intersection of $g$ and the image of the dashed circle.
Since the radius of the circle of inversion is arbitrary, and choosing another radius corresponds to scaling the inverted picture, any well-defined quantity derived from the inverted picture has to be invariant under scaling. Therefore, it has to be defined as an expression of the ratios of the concentric circles - which is exactly what Coxeter proceeds to do. He defines $(\alpha, \beta)$ - the "distance" of the two circles - to be the logarithm of the ratios of the radii of the inverted circles.