$\int_0^{\frac{\pi }{2}} \frac{\sin ^{2 m-1}(\theta ) \cos ^{2 n-1}(\theta )}{\left(a \sin ^2(\theta )+b \cos ^2(\theta )\right)^{m+n}} \, d\theta$ I tried substituting $x = \sin^2{\theta}$ then $dx = 2 \sin{\theta} \cos{\theta}$
Which got me to:
$\frac{1}{2}\int_0^1 \frac{x^{m-1} (1-x)^{n-1}}{ (a x+b (1-x))^{m+n}} \, dx$
But I have no idea how to proceed from here.
 A: $\newcommand{\bbx}[1]{\,\bbox[8px,border:1px groove navy]{\displaystyle{#1}}\,}
 \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
 \newcommand{\dd}{\mathrm{d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
 \newcommand{\ic}{\mathrm{i}}
 \newcommand{\mc}[1]{\mathcal{#1}}
 \newcommand{\mrm}[1]{\mathrm{#1}}
 \newcommand{\pars}[1]{\left(\,{#1}\,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
 \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
 \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$

Multiply the integrand numerator and denominator by
  $\ds{\sec^{2m + 2n}\pars{\theta}}$:

\begin{align}
& \int_{0}^{\pi/2}{\sin^{2m - 1}\pars{\theta}\cos^{2n - 1}\pars{\theta}
\over \bracks{a\sin^{2}\pars{\theta} + b\cos^{2}\pars{\theta}}^{\,m + n}}
\,\dd\theta
\\[5mm] = &\
\int_{0}^{\pi/2}{\bracks{\sec^{2m - 1}\pars{\theta}\sin^{2m - 1}\pars{\theta}}
\bracks{\sec^{2n - 1}\pars{\theta}\cos^{2n - 1}\pars{\theta}} \over \braces{\sec^{2}\pars{\theta}\bracks{a\sin^{2}\pars{\theta} + b\cos^{2}\pars{\theta}}}^{\,m + n}}\,\sec^{2}\pars{\theta}\,\dd\theta
\\[5mm] = &\
\int_{0}^{\pi/2}{\tan^{2m - 1}\pars{\theta} \over
\bracks{a\tan^{2}\pars{\theta} + b}^{\,m + n}}\,\sec^{2}\pars{\theta}\,\dd\theta 
\,\,\,\stackrel{x\ =\ \tan\pars{\theta}}{=}\,\,\,
\int_{0}^{\infty}{x^{2m - 1} \over \pars{ax^{2} + b}^{\,m + n}}\,\dd x
\\[5mm] \stackrel{x^{2}\ \mapsto\ x}{=}\,\,\,&\ 
{1 \over 2}\int_{0}^{\infty}{x^{m - 1} \over \pars{ax + b}^{\,m + n}}\,\dd\theta
=
\bbx{\ds{{1 \over 2a^{m}\,b^{n}}\,{\Gamma\pars{m}\Gamma\pars{n} \over \Gamma\pars{m + n}}}}
\end{align}

The last integral can be expressed in terms of an 
  integral representation of the Beta Function.

A: As Felix showed this integral is a version of Euler's beta function. 
Another way to derive that is to substitute $x = y/(1+y)$ in the equation you have.
We'll show a method to solve the more general integral. I've used the last formula we arrive at to solve your equation and another. 
$$F(p,q,r,x) = \int_0^x \frac{\cos^{p} x \sin^{q} x}{(a \cos^2 x + b \sin^2 x)^r} \mathbb{d}x $$
Note
$$\frac{\partial F(p,q,r,x)}{\partial a} = -rF(p+2,q,r+1,x) \\ 
\frac{\partial F(p,q,r,x)}{\partial b} = -rF(p,q+2,r+1,x)$$ 
Using $F(p+2,q,r,x) + F(p,q+2,r,x) = F(p,q,r,x) $
$$\frac{-1}{r}\bigg\{\frac{\partial F(p,q,r,x)}{\partial a} + \frac{\partial F(p,q,r,x)}{\partial b}\bigg\} = F(p,q,r+1,x)$$
and repeated application gives 
$$\phi_1(s)F(p,q,r,x)=\frac{(-1)^s(r-1)!}{(r+s-1)!}\bigg\{\frac{\partial}{\partial a} + \frac{\partial}{\partial b}\bigg\}^sF(p,q,r,x) = F(p,q,r+s,x) $$
Similarly reasoning shows
$$\phi_2(t)F(p,q,r,x)=\frac{(-1)^t(r-1)!}{(r+t-1)!}\bigg\{\frac{\partial}{\partial a}\bigg\}^tF(p,q,r,x)=F(p+2t,q,r+t,x)\\
\phi_3(v)F(p,q,r,x)=\frac{(-1)^v(r-1)!}{(r+v-1)!}\bigg\{\frac{\partial}{\partial b}\bigg\}^vF(p,q,r,x)=F(p,q+2v,r+v,x) $$
Hence a general method consists of integrating for as small of values of $p$ and $q$ as possible, call these values $c_p$ and $c_q$ and the integral $F(c_p,c_q,1,x)$ (using the process again if that is not clearly integrable), and successively applying operators $\phi_1,\phi_2,\phi_3$ to increase to the desired $p,q,r$. Doing this process shows
$$F(p,q,r,x)=\phi_1(s)\phi_2(t)\phi_3(v)F(c_p,c_q,1,x) = F(c_p+2t,c_q+2v,1+s+t+v,x)\\
= \boxed{\frac{(-1)^{r-1}}{(r-1)!}\bigg\{\frac{\partial}{\partial a}+\frac{\partial}{\partial b}\bigg\}^s \bigg\{\frac{\partial}{\partial a}\bigg\}^t \bigg\{\frac{\partial}{\partial b}\bigg\}^v F(c_p,c_q,1,x)}   $$
Making variables in each $F$ be equal shows 
$$s = r-1-t-v \\ t = (p-c_p)/2 \\ v=(q-c_q)/2$$ 
where $s\geq 0$ and $s,t,v \in \mathbb{N}$ and $c_p, c_q$ are to be as small as allowed so as to make the initial integral $F(c_p,c_q,1,x)$ simpler. 
For certain $F(c_p,c_q,1,x)$ or small $p,q,r$ this equation may be entirely manageable. 
