I am trying to find for $a>b>0$ : $$I=\int_{0}^{\pi }{\frac{x\sin \left( x \right)}{a+b{{\cos }^{2}}\left( x \right)}dx}$$ I don't think a substitution is an option here, besides differentiation under the sign integral makes the integral harder. Fortunately Mathematica gives a closed form for the integral in question:
$$\frac{\pi \tan^{-1}\left( \sqrt{\frac{b}{a}} \right)}{\sqrt{ab}}$$
HINT:
Enforce the substitution $x\mapsto \pi-x$.
Then, note that
$$I=\int_0^\pi \frac{x\sin(x)}{a+b\cos^2(x)}\,dx=\int_0^\pi \frac{(\pi-x)\sin(x)}{a+b\cos^2(x)}\,dx$$
Can you finish now?
$\newcommand{\bbx}[1]{\,\bbox[15px,border:1px groove navy]{\displaystyle{#1}}\,} \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace} \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack} \newcommand{\dd}{\mathrm{d}} \newcommand{\ds}[1]{\displaystyle{#1}} \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,} \newcommand{\ic}{\mathrm{i}} \newcommand{\mc}[1]{\mathcal{#1}} \newcommand{\mrm}[1]{\mathrm{#1}} \newcommand{\pars}[1]{\left(\,{#1}\,\right)} \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}} \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,} \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}} \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$ \begin{align} I & \equiv \int_{0}^{\pi}{x\sin\pars{x} \over a + b\cos^{2}\pars{x}}\,\dd x = \int_{-\pi/2}^{\pi/2}{\pars{x + \pi/2}\cos\pars{x} \over a + b\sin^{2}\pars{x}}\,\dd x \\[5mm] & = \pi\int_{0}^{\pi/2}{\cos\pars{x} \over a + b\sin^{2}\pars{x}}\,\dd x = \pi\int_{0}^{1}{\dd x \over bx^{2} + a} \\[5mm] & = \pi\,{1 \over a}\,\root{a \over b}\int_{0}^{1}{\root{b/a}\dd x \over \pars{\root{b/a}x}^{2} + 1} = {\pi \over \root{ab}}\int_{0}^{\root{b/a}}{\dd x \over x^{2} + 1} \\[5mm] & = \bbx{{\pi \over \root{ab}}\,\arctan\pars{\root{b \over a}}} \end{align}
