Prove that $\int^{\pi}_0 \ln \left(\frac{b-\cos x}{a- \cos x}\right)dx=\pi \ln (\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}})$ Given for $\alpha>1$
$$\int^{\pi}_0 \frac{dx}{\alpha-\cos x}=\frac{\pi}{\sqrt{\alpha^2-1}}$$
Prove that $$\int^{\pi}_0 \ln \left(\frac{b-\cos x}{a- \cos x}\right)dx=\pi \left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right)$$
if $a, b>1$
 A: First, notice the integrand can be written as
$$
\begin{align}
\ln (\frac{b-\cos x}{a- \cos x}) &=\ln(b-\cos x)-\ln(a-\cos x)\\
& =\ln(y-\cos x)|^{y=b}_{y=a}\\
& =\int^b_a\frac{1}{y-\cos x}dy
\end{align}
$$
So the integration equals 
$$
\begin{align}
\int^{\pi}_0 \ln (\frac{b-\cos x}{a- \cos x})dx &=\int^{\pi}_0 [ \int^b_a\frac{1}{y-\cos x}dy]dx \\ 
&=\int^b_a [\int^{\pi}_0 \frac{1}{y-\cos x}dx ] dy\\
& =\int^b_a(\frac{\pi}{\sqrt{y^2-1}})dy
\end{align}
$$
Let $$y=\sec t$$
so 
$$dy=\sec t\tan t dt$$
So the integrantion becomes
$$
\begin{align}
\pi \int^{acrsec b}_{arcsec a}\frac{\sec t\tan t}{\tan t}dt &=\pi \int^{acrsec b}_{arcsec a}\sec t dt\\ & = \pi \ln |\sec t+\tan t|^{arcsec b}_{arcsec a}\\ & = \pi [\ln(b+\sqrt{b^2-1})-\ln(a+\sqrt{a^2-1})] \\ & =\pi \ln (\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}})
\end{align} 
$$
A: 
Since one posted solution relied on, and one comment discussed the real analysis methodology of differentiating under the integral, I thought it might be instructive to present a way forward that relies on contour integration.  To that end, we proceed.


Let $I(a,b)$ be the integral given by
$$\begin{align}
I(a,b)&=\int_0^\pi \log\left(\frac{b-\cos(x)}{a-\cos(x)}\right)\,dx\\\\
&=\frac12 \int_0^{2\pi} \log\left(\frac{b-\cos(x)}{a-\cos(x)}\right)\,dx \\\\
&\frac12 \int_0^{2\pi }\left(\log(b-\cos(x))-\log(a-\cos(x))\right)\,dx\tag 1
\end{align}$$
We will evaluate the integral in $(1)$ by moving to the complex plane.  We let $z=e^{ix}$ in $(1)$ to arrive at
$$\begin{align}
I(a,b)&=\frac12 \oint_{|z|=|}\frac {1}{iz} \left(\log(z-r_{b+})-\log(z-r_{a+})\right)\,dz \\\\
&+\frac12\oint_{|z|=1}\frac {1}{iz} \left(\log(z-r_{b-})-\log(z-r_{a-})\right)\,dz \tag 2
\end{align}$$
where $r_{a\pm}=a\pm\sqrt{a^2-1}$ and $r_{b\pm}=b\pm\sqrt{b^2-1}$.  


Note that the branch points at $r_{a+}$ and $r_{b+}$ are not enclosed by the unit circle.  Hence, the integrand of the first integral on the right-hand side of $(2)$ is meromorphic for $|z|\le 1$, with a simple pole at $z=0$.

Applying the residue theorem to the first integral on the right-hand side of $(2)$, we obtain 
$$\begin{align}
\frac12 \oint_{|z|=1}\frac {1}{iz} \left(\log(z-r_{b+})-\log(z-r_{a+})\right)\,dz&=\frac12 \left(2\pi i \text{Res}\left(\frac {1}{iz} \left(\log(z-r_{b+})-\log(z-r_{a+})\right),z=0\right)\right) \\\\
&=\frac12\left(2\pi i \frac1i \log\left(\frac{r_{b+}}{r_{a+}}\right) \right)\\\\
&=\pi \log\left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right) \tag 3
\end{align}$$


On the other hand, the branch points at $z=r_{a-}$ and $z=r_{b-}$ are enclosed by the unit circle.  We can cut the plane so that a suitable branch cut is the segment that adjoins $r_{a-}$ and $r_{b-}$.  Then, to evaluate the second integral on the right-hand side of $(2)$, we note that the integrand is analytic in the annular region $1\le |z|\le R$ for any $R>1$.  

Therefore, Cauchy's Integral Theorem guarantees that 
$$\begin{align}
\oint_{|z|=1}\frac iz \left(\log(z-r_{b-})-\log(z-r_{a-})\right)\,dz&=\lim_{R\to \infty}\oint_{|z|=R}\frac iz \left(\log(z-r_{b-})-\log(z-r_{a-})\right)\,dz\\\\
&=\lim_{R\to \infty}\int_0^{2\pi}\frac{i}{Re^{i\phi}}\left(\log\left(1-\frac{r_{b-}}{Re^{i\phi}}\right)-\log\left(1-\frac{r_{a-}}{Re^{i\phi}}\right)  \right)\,iRe^{i\phi}\,d\phi\\\\
&=0 \tag 4
\end{align}$$

Finally, putting together $(3)$ and $(4)$ yields the coveted result

$$\bbox[5px,border:2px solid #C0A000]{\int_0^\pi \log\left(\frac{b-\cos(x)}{a-\cos(x)}\right)\,dx=\pi \log\left(\frac{b+\sqrt{b^2-1}}{a+\sqrt{a^2-1}}\right)}$$

