Compute $\int_0^\pi\frac{\cos nx}{a^2-2ab\cos x+b^2}\, dx$ 
How to compute the following integral
  \begin{equation}
\int_0^\pi\frac{\cos nx}{a^2-2ab\cos x+b^2}\, dx
\end{equation}

I have been given two integral questions by my teacher. I cannot answer this one. I have also searched the similar question here but it looks like nothing is similar so I think this is not a duplicate. I could compute the integral if 
\begin{equation}
\int_0^\pi\frac{dx}{a^2-2ab\cos x+b^2}
\end{equation}
The $\cos nx$ part makes the integral is really difficult. I want to use the result to compute this integral (the real question given by my teacher)
\begin{equation}
\int_0^\pi\frac{x^2\cos nx}{a^2-2ab\cos x+b^2}\, dx
\end{equation}
My question is how to compute the first integral (in the grey-shaded part) preferably with elementary ways (high school methods)? 
 A: I don't have a proof yet, but with some computer assistance, this is what I believe to be the answer when $n$ is a non-negative integer.
\begin{align*}
  \int_0^\pi\frac{\cos nx}{a^2-2ab\cos x+b^2}\, dx &= \frac{\displaystyle \left(\left(a^{2n}+b^{2n}\right)\, \left(\Big\lvert \frac{a+b}{a-b}\Big\rvert +1\right)-2\, \sum_{k=0}^{2n}a^k\, b^{2n-k}\right)\, \pi}{\displaystyle 2\, a^n\, b^n\, (a+b)^2}
\end{align*}
I think there might be a reduction formula for this, but at the moment I don't know.
A: Let $I_n(a,b)$ be the desired integral. Note that $I_n(a,b)=I_n(b,a)$, and $I_n(a,b)=I_n(-a,-b)$. So, we may suppose that $|b|< a$ Note that
$$\eqalign{
\frac{a^2-b^2}{a^2-2ab\cos x+b^2}&=\frac{a}{a-e^{ix}b}+\frac{be^{-ix}}{a-e^{-ix}b}\cr
&=\sum_{n=0}^\infty \left(\frac{b}{a}\right)^ne^{inx}+\frac{be^{-ix}}{a}\sum_{n=0}^\infty
\left(\frac{b}{a}\right)^ne^{-inx}\cr
&=1+\sum_{n=1}^\infty \left(\frac{b}{a}\right)^ne^{inx}+ \sum_{n=1}^{\infty}
\left(\frac{b}{a}\right)^{n}e^{-inx}\cr
&=1+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\cos(n x)
}
$$
It follows, using the uniform convergence of the series on $[0,\pi]$, that
$$
\int_0^\pi\frac{(a^2-b^2)\cos(mx)}{a^2-2ab\cos x+b^2}dx
=\int_0^\pi\cos(mx)dx+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\int_0^\pi\cos(n x)\cos(mx)dx
$$
But $\int_0^\pi\cos(n x)\cos(mx)dx=0$ if $n\ne m$, and 
$\int_0^\pi\cos^2(n x)dx=\pi/2$ if $n\ne0$. So
$$\eqalign{I_m(a,b)=
\int_0^\pi\frac{\cos(mx)}{a^2-2ab\cos x+b^2}dx
&=\left\{\matrix{\frac{\pi}{a^2-b^2}&\hbox{if}&m=0\cr
\frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^m&\hbox{if}&m\ne0
}
\right.}
$$
which is the desired formula for $|b|<a$.
A: Assume for definiteness that $a>b>0$. 
Method 1:
For integer $n\geq0$, we can rewrite the integral as
$$\frac{1}{4ab}\int_{-\pi}^{\pi}\frac{e^{in x}dx}{\cosh\gamma-\cos x}=\frac{1}{2ia^2}\oint_{|z|=1}\frac{z^{n}dz}{(z-e^{-\gamma})(1-e^{-\gamma}z)},$$
where $e^{\gamma}=\frac{a}{b}$. Computing the residue at $z=e^{-\gamma}$, we find for the last integral
$$2\pi i \cdot \frac{1}{2ia^2}\cdot\frac{\left(\frac{b}{a}\right)^n}{1-\frac{b^2}{a^2}}=\frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^n.$$
Method 2:
Similarly  rewrite the integral as
$$\frac{1}{2(a^2-b^2)}\int_{-\pi}^{\pi}\left(\frac{1}{1-\frac{b}{a}e^{ix}}+\frac{\frac{b}{a}e^{-ix}}{1-\frac{b}{a}e^{-ix}}\right)e^{in x}dx.$$
Then expand the integrand into series in $\frac{b}{a}$ and use that $\displaystyle\int_{-\pi}^{\pi}e^{inx}dx=2\pi\cdot\delta_{n,0}$.
A: See the last comment in
Evaluating $\int_0^{\pi}\frac{\cos n\eta\,d\eta}{1+a^2+2a\cos m\eta}$
for the answer to your original `$x^2\cos nx$' integral which I don't think is answered in any of the above posts.
