$\int_a^b \frac{(x-a)^p(b-x)^{1-p}}{\text{P}(x)} dx$ using contour integration and residue theorem So a while ago I came across these types of integrals that has the form of
$$
I = \int_a^b (x-a)^p(b-x)^{1-p}dx
$$
Where $a$, $b$ and $p$ are real numbers, $a<b$ and $0<p<1$. And those types integral has a closed form, namely
$$
I = \frac{\pi}{2}p(1-p)(b-a)^2\csc(p\pi)
$$
But then I also came across integrals like these along the way
$$
\int_0^3 \frac{x^{3/4}(3-x)^{1/4}}{5-x}dx
$$
So I asked myself, is there a more generalized form of the $I$ integral, namely
$$
J = \int_a^b \frac{(x-a)^p(b-x)^{1-p}}{\text{P}(x)} dx
$$
Where $\text{P}(x)$ is any polynomial.
To begin with, I decided to make a more concise (?) definition of the function
$$
\text{P}(x) = \prod_{k=1}^n (x-c_k)^{q_k}
$$
Where $c_k$ is the $k$-th root of $\text{P}(x)$ and $q_k$ is the $k$-th root's multiplicity. $c_1 \not = c_2 \not = c_3 \not = \ldots \not = c_n$, $c_k \not \in (a, b)$, and if $a$ or $b$ is a root of $\text{P}(x)$ then they can't have a multiplicity of 2 or higher
First off, I'll analyze this function
$$
f(z) = \frac{(z-a)^p (b-z)^{1-p}}{\text{P}(z)} \\
= \frac{|z-a|^p |b-z|^{1-p}}{\text{P}(z)} e^{i(p\arg(z-a) + (1-p)\arg(b-z))}
$$
The poles of this function are $z = c_k$ with order $q_k$ for $k = 1,2,3,\ldots, n$
To begin, I integrated $f(z)$ over the dogbone contour

Which means I'll have
$$
\oint_{\text{Dogbone}} f(z) dz = \left(\int_{\gamma_1} + \int_{\gamma_2} + \int_{\mu_1} + \int_{\mu_2}\right) f(z) dz
$$
If we call the radius of the $\mu$ paths $\varepsilon$, and have it approach 0, then both integrals over $\mu_1$ and $\mu_2$ will go to 0 also. The integral over $\gamma_1$ will be $J$ and the integral over $\gamma_2$ will be $-e^{(1-p)2\pi} J$.
Which means
$$
\oint_{\text{Dogbone}} f(z) dz = \left(1 - e^{(1-p)2\pi}\right) J
$$
Now, to continue, we can inflate the contour into this

I'll call the big circular contour $\Gamma$ and the small ones surrounding the poles at $c_k$ $\omega_k$. Which means we'll have
$$
\left(1 - e^{(1-p)2\pi}\right) J = \left(\oint_\Gamma + \sum_{k=1}^n \oint_{\omega_k}\right) f(z) dz
$$
For the $\omega$ integrals, we can simply use residue theorem for higher order poles
$$
\oint_{\omega_k} f(z) dz = - \frac{2\pi i}{(q_k - 1)!} \lim_{z \rightarrow c_k} \partial_z^{q_k - 1} (z-c_k)^{q_k} f(z)
$$
Plugging it back into the equation
$$
\left(1 - e^{(1-p)2\pi}\right) J = \oint_\Gamma f(z) dz - \sum_{k=1}^n \frac{2 \pi i}{(q_k - 1)!} \lim_{z \rightarrow c_k} \partial_z^{q_k - 1} (z-c_k)^{q_k} f(z)
$$
Now, my main problem is the final integral. I know that it'll be the residue of $f(z)$ at infinity, I know there's a formula for it, but I have no idea of how to evaluate such residues. My guess is that it'll some how evaluate to 0, but still, I have no idea how that'll be the answer.

26/2 Edit: Fixed some of the mistakes in calculation + Thanks to @Diger for pointing out the mistake in the inflated contour
 A: $\newcommand{\d}{\,\mathrm{d}}\newcommand{\res}{\operatorname{Res}}$Define $z\mapsto z^p$ via the principal logarithm $(-\pi\le\arg z<\pi)$ and $z\mapsto z^{1-p}$ via the logarithm induced by $0\le\arg<2\pi$. Let's call these two argument functions $\arg_1,\arg_2$ respectively.
Let's take a polynomial $\lambda$ which has no roots on $[a,b]$ (no assumptions other than that). $p$ is in $(0,1)$ and it is important that $b>a\ge0$. The case $a=0$ requires special attention in the final case for linear $\lambda$.
Then: $f:\Bbb C\to\hat{\Bbb C}$, $z\mapsto(z-a)^p(b-z)^{1-p}\lambda(z)^{-1}$ is meromorphic outside of $[a,b]$. This depends in an essential way on $\arg_1(-x)=-\pi$ when $x$ is a positive real and on the combination $p\arg_1+(1-p)\arg_2$.
You want to know the value of $\res(f,\infty)$. Assuming the rest of your work is correct, knowledge of this will give you the value of your integral.
Let $R$ be a real number larger than the modulus of every root of $\lambda$.
If $|z|\ge R$ then $|f|=\frac{|z|}{|\lambda(z)|}\cdot|1-a/z|^p\cdot|1-b/z|^{1-p}$. If $\lambda$ has degree greater than two, then: $$|z\cdot f(z)|\overset{|z|\to\infty}{\longrightarrow}0$$
Which is a sufficient condition for $\res(f,\infty)=0$, so you get to ignore this term
More generally, if $f$ is holomorphic on some large annulus and vanishes as $|z|\to\infty$ then $\lim_{|z|\to\infty}z\cdot f(z)=\res(f,\infty)$. This is justified by the "series expansion at infinity". We can apply this when the degree of $\lambda$ is two: say, $\lambda(z)=\alpha\cdot z^2+\beta\cdot z+\gamma$ where $\alpha\neq0$.
Then: $$\begin{align}\res(f,\infty)&=\lim_{|z|\to\infty}\frac{z\cdot |z|}{\alpha\cdot z^2+\beta\cdot z+\gamma}|1-a/z|^p|1-b/z|^{1-p}e^{ip\arg_1(z-a)+i(1-p)\arg_2(b-z)}\\&=\lim_{|z|\to\infty}\frac{1}{\alpha\cdot\frac{z}{|z|}}e^{ip\arg_1(z-a)+i(1-p)\arg_2(b-z)}\end{align}$$
If $z$ is on the negative real axis, the expression is $\frac{1}{-\alpha}e^{-i\pi p}$. This is cheating really, but since the limit must exist, it in particular must equal the value as $|z|\to\infty$ in the negative real axis so we can conclude: $$\res(f,\infty)=-\frac{e^{-\pi ip}}{\alpha}$$
Finally, say $\lambda(z)=\alpha\cdot z+\beta$ is linear, where $\alpha\neq0$. Then we need a different approach, as $f$ no longer vanishes as $|z|\to\infty$. The definition $\res(f,\infty)=-\res(f(1/z)/z^2,0)$ becomes useful now.

There are some subtle aspects in the evaluation below. I casually pretend the things we are used to from real analysis apply for complex numbers, but this is not so. I mention the subtleties at the end.

If $\gamma:=\beta/\alpha$, then: $$\frac{1}{z^2}f(1/z)=\frac{e^{\pi i(1-p)}}{\alpha\cdot z^2}\cdot\frac{1}{1+\gamma z}(1-az)^p(1-bz)^{1-p}$$
Expand in series: $$\frac{1}{z^2}f(1/z)=-\frac{e^{-i\pi p}}{\alpha\cdot z^2}\cdot(1-\gamma z+o(z))(1-paz+o(z))(1-(1-p)bz+o(z))$$
We want to extract the $z^{-1}$ coefficient: $$-\frac{e^{-i\pi p}}{\alpha}(-\gamma-pa-(1-p)b)$$
Concluding: $$\res(f,\infty)=-\frac{e^{-i\pi p}}{\alpha}\cdot(\gamma+b-p(b-a))$$

NOTE: I used the equalities $(1-az)^p/z^p=((1/z)-a)^p$, $(bz-1)^{1-p}/z^{1-p}=(b-(1/z))^{1-p}$ and $(1/z^p)\cdot(1/z^{1-p})=1/z$. These are actually quite subtle: so subtle, that they are not even always correct. However, I leave it as an exercise in geometric reasoning to show that the product of all these ‘equations’ is a genuine equation for small $z$.
Expanding in binomial series is also subtle. I presented it in the simplest, but technically misleading, way. So long as $|z|<1/a$, $(1-az)^p$ is holomorphic and we can do the usual Taylor expansion. But for small $z$, $(1-bz)^{1-p}$ is not holomorphic. But, $(bz-1)^{1-p}$ is holomorphic for small $z$ and we get the usual Taylor series *multiplied by a factor of $(-1)^{1-p}=-e^{-i\pi p}$. Phew!
A: I once wrote down the solution for $n=1$, but it should be generalizable. $m=1$ in your case.
Find the closed form for
\begin{align*}
I(x)=\frac{\Gamma(p+1)\Gamma(m-p+1)}{\Gamma(m+2)} \, {\rm _2F_1}\left(n,p+1;m+2;x\right)=\int_0^{1} \frac{t^p\left(1-t\right)^{m-p}}{\left(1-xt\right)^n} \, {\rm d}t \, ,
\end{align*}
with $m,n\in\mathbb{N}$, $\Re(p)\in(-1,m+1)$ and $x\in D_1(0)$.
solution: We start by considering the integral
\begin{align}
J(x)=\oint_C \frac{z^p (1-z)^{m-p}}{(1-xz)^n} \, {\rm d}z \tag{1}
\end{align}
where $C=C_1+\gamma_1+C_2+\gamma_2$ is the dog-bone contour with $C_1=(-i\epsilon,1-i\epsilon)$, $\gamma_1=\{z\in\mathbb{C}:|z-1|=\epsilon \land \Re(z-1)\geq 0\}$, $C_2=(1+i\epsilon,i\epsilon)$, $\gamma_2=\{z\in\mathbb{C}:|z|=\epsilon \land \Re(z)\leq 0\}$. The integrand has cuts and poles in the complex plane, but we are free to choose the discontinuity of $z\mapsto z^p$ to be $(-\infty,0)$ and that of $z\mapsto z^{m-p}$ to be $(0,\infty)$, which means $z\mapsto (1-z)^{m-p}$ has a cut on $(-\infty,1)$. Furthermore the denominator leads to poles of order $n$ at $z=1/x$ and has no cuts. We may also assume $1/x\notin \bar{D}_1(0)$. In order to determine the cut of the integrand, we need to analyze the combined behaviour of $z^p(1-z)^{m-p}$ by considering the phase of this expression as we cross the three segments $(-\infty,0)$, $(0,1)$ and $(1,\infty)$. Writing $z=t\pm i0$, we find
\begin{align*}
t\in(-\infty,0) : \quad \arg\left( (t\pm i0)^p(1-t\mp i0)^{m-p} \right) &= \begin{cases} \pi p + 2\pi(m-p) \quad &\text{if} \quad z=t+i0 \\ -\pi p + 0 \quad &\text{if} \quad z=t-i0 \end{cases} \\
t\in(0,1) : \quad \arg\left( (t\pm i0)^p(1-t\mp i0)^{m-p} \right) &= \begin{cases} 0 + 2\pi(m-p) \quad &\text{if} \quad z=t+i0 \\ 0 + 0 \quad &\text{if} \quad z=t-i0 \end{cases} \\
t\in(1,\infty) : \quad \arg\left( (t\pm i0)^p(1-t\mp i0)^{m-p} \right) &= \begin{cases} 0 + \pi(m-p) \quad &\text{if} \quad z=t+i0 \\ 0 + \pi(m-p) \quad &\text{if} \quad z=t-i0 \end{cases}
\end{align*}
and since multiples of $2\pi$ leave the argument invariant, the only case that leads to a discontinuity is $t\in(0,1)$. Furthermore, for the given values of $p$, it is easy to see that the integrals over $\gamma_1$ and $\gamma_2$ vanish in the limit $\epsilon \rightarrow 0$. Then continuing with Equation (1), it follows
\begin{align}
\notag J(x) &= \int_{-i0}^{1-i0} \frac{z^p (1-z)^{m-p}}{(1-xz)^n} \, {\rm d}z + \int_{1+i0}^{i0} \frac{z^p (1-z)^{m-p}}{(1-xz)^n} \, {\rm d}z \\
\notag &= \int_{0}^{1} \frac{t^p (1-t)^{m-p}}{(1-xt)^n} \, {\rm d}t - e^{-i2\pi p} \int_{0}^{1} \frac{t^p (1-t)^{m-p}}{(1-xt)^n} \, {\rm d}t \\
&= \left(1 - e^{-i2\pi p} \right) I(x) \tag{2} \, .
\end{align}
On the other hand, we may now solve Equation (1) by the residue theorem. Since $1/x$ is outside the unit disk, we can deform $C$ to the unit circle $|z|=1$ and substitute $z=1/z'$
\begin{align}
J(x)=\oint_{|z'|=1} \frac{z'^{-p-2} (1-1/z')^{m-p}}{(1-x/z')^n} \, {\rm d}z' = \oint_{|z|=1} \frac{z^{n-m-2} (z-1)^{m-p}}{(z-x)^n} \, {\rm d}z \tag{3}
\end{align}
where a minus sign was absorbed to reverse the orientation of the contour back to run counter-clockwise. We can now see, that the cut on the RHS of Equation (3) is outside the unit disk and we can use the residue theorem for the singularities at $z=0$ and $z=x$ inside the unit circle
\begin{align}
 J(x)=2\pi i \, \underbrace{{\rm Res} \left( \frac{z^{n-m-2} (z-1)^{m-p}}{(z-x)^n} \right)_{z=0}}_{=R_0} + 2\pi i \, \underbrace{{\rm Res} \left( \frac{z^{n-m-2} (z-1)^{m-p}}{(z-x)^n} \right)_{z=x}}_{=R_x} \, .
\end{align}
For the residue at $z=0$ we find
\begin{align*}
 R_0&=\frac{e^{i\pi(n+m-p)}}{x^n} \, {\rm Res} \left( \frac{z^{n-m-2}(1-z)^{m-p}}{(1-z/x)^n} \right)_{z=0} \\
 &=\frac{e^{i\pi(n+m-p)}}{x^n} \, {\rm Res} \left( z^{n-m-2} \sum_{k,l=0}^\infty \binom{m-p}{k}\binom{-n}{l} (-z)^k (-z/x)^l \right)_{z=0} \\
 &= \frac{e^{-i\pi p}}{x^n} \sum_{\substack{k,l=0 \\ k+l+n-m-2=-1}}^\infty \binom{m-p}{k}\binom{-n}{l} (-1)^{k+l+n+m} x^{-l} \\
 &= -e^{-i\pi p} \sum_{\substack{l=0}}^{m+1-n} \binom{m-p}{m+1-n-l}\binom{-n}{l} x^{-l-n}
\end{align*}
and similarly at $z=x$
\begin{align*}
 R_x&=\frac{1}{(n-1)!} \frac{{\rm d}^{n-1}}{{\rm d}z^{n-1}} \, z^{n-m-2} (z-1)^{m-p} \bigg|_{z=x} \\
 &= \frac{1}{(n-1)!} \sum_{k=0}^{n-1} \binom{n-1}{k} \left( \frac{{\rm d}^{n-1-k}}{{\rm d}x^{n-1-k}} \, x^{n-m-2} \right) \left( \frac{{\rm d}^{k}}{{\rm d}x^{k}} \, (x-1)^{m-p} \right) \\
 &=\sum_{k=0}^{n-1} \binom{n-m-2}{k-m-1} \binom{m-p}{k} x^{k-m-1} (x-1)^{m-p-k} \\
 &=-e^{-i\pi p}\sum_{k=0}^{n-1} \binom{n-m-2}{k-m-1} \binom{m-p}{k} (-x)^{k-m-1} (1-x)^{m-p-k} \, .
\end{align*}
With Equation (2) this finally gives
\begin{align*}
 I(x)&=-\frac{\pi}{\sin(\pi p)} \sum_{\substack{l=0}}^{m+1-n} \binom{m-p}{m+1-n-l}\binom{-n}{l} x^{-l-n} \\
 &\quad - \frac{\pi}{\sin(\pi p)} \sum_{k=0}^{n-1} \binom{n-m-2}{k-m-1} \binom{m-p}{k} (-x)^{k-m-1} (1-x)^{m-p-k} \, .
\end{align*}
