A little complex analysis helps. First, by the evenness of $\cos$, we have
$$I_n = \frac12 \int_{-\pi}^\pi \frac{\cos (nx)}{a+\cos x}\,dx,$$
and by Euler's formula, we have $\cos (nx) = \operatorname{Re} e^{inx}$, so
$$I_n = \frac12 \operatorname{Re} \int_{-\pi}^\pi \frac{e^{inx}}{a+\cos x}\,dx.$$
Since the imaginary part is odd, the integral is real, and we can omit the $\operatorname{Re}$.
Now we write $z = e^{ix}$, then we have
$$\cos x = \frac12(z+z^{-1});\quad e^{inx} = z^n;\quad dx = \frac{dz}{iz};$$
so we obtain
$$I_n = \int_{\lvert z\rvert = 1} \frac{z^n}{2a + z + z^{-1}}\, \frac{dz}{iz} = 2\pi\cdot\frac{1}{2\pi i}\int_{\lvert z\rvert = 1} \frac{z^n}{z^2 + 2az + 1}\,dz.$$
We need the zeros of the denominator $z^2 + 2az + 1 = (z+a-\sqrt{a^2-1})(z+a+\sqrt{a^2-1})$ to apply the Cauchy integral formula. Of the zeros, $-a+\sqrt{a^2-1}$ lies inside the unit disk, and $-a-\sqrt{a^2-1}$ outside the closed unit disk. With $f(z) = \frac{z^n}{z+a+\sqrt{a^2-1}}$, the Cauchy integral formula yields
$$\begin{align}
I_n &= 2\pi\cdot\frac{1}{2\pi i}\int_{\lvert z\rvert = 1} \frac{f(z)}{z+a-\sqrt{a^2-1}}\,dz\\
&= 2\pi\cdot f(-a+\sqrt{a^2-1})\\
&= \frac{\pi}{\sqrt{a^2-1}}(\sqrt{a^2-1}-a)^n.\tag{1}
\end{align}$$