How to prove that $X, Y,$ and $Z$ are collinear if and only if $\cos A \cos B \cos C =\frac{-3}{8}$? Let $X, Y, Z$ be the reflections of $A, B$, and $C$ across the lines $BC, CA,$ and $AB$, respectively. How to prove that $X, Y,$ and $Z$ are collinear if and only if $\cos A \cos B \cos C =\frac{-3}{8}$? for example α=120, β=γ=30.

 A: In order to obtain manageable formulas we have to destroy the inherent  symmetries of the problem.
Represent $A$, $B$, $C$ as complex numbers with $A$ at the origin and $B\vee C$ horizontal as in your figure. Then, up to similarity, we have
$$A=0,\quad B=-\sin\gamma\> e^{-i\beta},\quad C=\sin\beta\> e^{i\gamma}$$
and
$$X=2i\>\sin\beta\>\sin\gamma,\quad Y=-\sin\gamma\>e^{i(\beta+2\gamma)},\quad Z=\sin\beta\>e^{-i(2\beta+\gamma)}\ .$$
The three points $X$, $Y$, $Z$ are collinear iff
$${X-Y\over Z-X}\in{\mathbb R},\qquad{\rm resp.},\qquad(X-Y)(\bar Z-\bar X)\in{\mathbb R}\ .$$
This amounts to
$$\sin\gamma\bigl(2i\sin\beta+e^{i(\beta+2\gamma)}\bigr) \sin\beta \bigl(e^{i(2\beta+\gamma)}+2i\sin\gamma\bigr)\in{\mathbb R}\ .\tag{1}$$
Here the factor $\sin\beta\sin\gamma>0$ can be omitted, so that $(1)$ can be rewritten in real terms as
$$\cos(\beta+2\gamma)\bigl(\sin(2\beta+\gamma)+2\sin\gamma\bigr)+\bigl(2\sin\beta+\sin(\beta+2\gamma)\bigr)\cos(2\beta+\gamma)=0\ .\tag{2}$$
Now Mathematica found out for me that the left hand side of $(2)$ can be factored into
$$\sin(\beta+\gamma)\bigl(2\cos(2\beta+2\gamma)+2\cos(2\beta)+2\cos(2\gamma)-1\bigr)\ .$$
Since $\sin(\beta+\gamma)\ne0$ we may conclude that $$\cos(2\alpha)+\cos(2\beta)+\cos(2\gamma)={1\over2}\ ,$$
which according to @Aretino 's   comment  is equivalent to your claim.
