Closed form for $\int_0^\alpha \int_0^{\alpha-x} \frac{1}{[(x+y)(\beta-x)(1-\beta-y)]^{3/2}} \,dy \,dx$ Let $0\leq \alpha \leq \frac{1}{2},$ and let $\beta$ be such that $\alpha \leq \beta \leq 1-\alpha$.
Define $\varphi(\alpha,\beta)$ to be the integral
$$\varphi(\alpha,\beta) = \int_0^\alpha \int_0^{\alpha-x} \frac 1 {[(x+y)(\beta-x)(1-\beta-y)]^{3/2}} \,dy\, dx.$$
It'd be nice to have some closed expression for this integral, but so far I've been unable to come up with one.
Mathematica starts to spit out some really large expressions involving lots of complex numbers, and I've been unable to convince it that the answer should be real.
Does anyone know of a substitution or other method that would yield a nicer expression?
 A: Let $z = x+y$, we have
$$\varphi(\alpha,\beta) = \int_0^\alpha \frac{dz}{z^{3/2}}\left\{\int_0^z \frac{dy}{((\beta-z+y)(1-\beta-y))^{3/2}}\right\}$$
For fixed $z$, introduce variables $u, v$ such that $u = \beta -z + y$, $v = 1 - \beta-y$. It is easy to check
$$d\left(\frac{u-v}{\sqrt{uv}}\right)
= \left(\frac{2}{\sqrt{uv}} - \frac{(u-v)(v-u)}{2\sqrt{uv}^3}\right) dy
= \frac{(u+v)^2}{2\sqrt{uv}^3} dy = \frac{(1-z)^2}{2\sqrt{uv}^3} dy
$$
and hence
$$\begin{align}
\varphi(\alpha,\beta) 
=& 2 \int_0^\alpha \frac{dz}{z^{3/2}(1-z)^2} \left\{
\left[\frac{u-v}{\sqrt{uv}}\right]_{y=0}^{y=z}
\right\}\\
=& 2 \int_0^\alpha \frac{dz}{z^{3/2}(1-z)^2}\left\{
\frac{2\beta-1 +z}{\sqrt{\beta(1-\beta-z)}} - \frac{2\beta-1-z}{\sqrt{(1-\beta)(\beta-z)}}
\right\}\\
=& -4\frac{\partial}{\partial\beta} \int_0^\alpha \frac{dz}{z^{3/2}(1-z)^2}\left\{
\sqrt{\beta(1-\beta-z)} - \sqrt{(1-\beta)(\beta-z)}
\right\}\\
=& -4\frac{\partial}{\partial\beta}\left[
\sqrt{\beta(1-\beta)}
\left( F\left(\frac{1}{1-\beta},\alpha\right) - F\left(\frac{1}{\beta},\alpha\right) \right)
\right]
\end{align}
$$
where
$$F(\mu,z) = \int^z \frac{\sqrt{1-\mu t}}{t^{3/2}(1-t)^2} dt + \text{ constant }.$$
To evaluate $F(\mu,z)$, let us introduce new variables
$$s = \frac{t}{1-\mu t} \;\;\leftrightarrow\;\; t = \frac{s}{1+\mu s}
\quad\implies\quad ds = \frac{dt}{(1-\mu t)^2}
$$
and let $\nu = \mu - 1$. Notice $0 < \beta < 1$ implies both
$\frac{1}{1-\beta}$ and $\frac{1}{\beta}$ are bigger than $1$. This means in this problem,
$\nu$ is always positive. In terms of these variables, we can transform $F(\mu,z)$ as
$$
F(\mu,z) 
= \int^z \frac{dt}{(1-\mu t)^2} \sqrt{\frac{1-\mu t}{t}}\left(\frac{1-\mu t}{t}\right)^2 \frac{t}{(1-t)^2}
= \int \frac{ds}{\sqrt{s}} \frac{1+\mu s}{s(1+\nu s)^2}
$$
Notice 
$$\begin{align} \frac{1+\mu s}{s(1+\nu s)^2}
=& \frac{(1+\nu s) + s}{s(1+\nu s)^2} 
= \frac{1}{s(1+\nu s)} + \frac{1}{(1+\nu s)^2}
= \frac{1}{s} - \frac{\nu}{1+\nu s} + \frac{1}{(1+\nu s)^2}\\
=& \frac{1}{s} + \left(\frac{\partial}{\partial \nu} - 1 \right) \frac{\nu}{1+\nu s}
= \frac{1}{s} + \left(\nu \frac{\partial}{\partial \nu} + 1 - \nu\right)\frac{1}{1 + \nu s}
\end{align}$$
We get
$$\begin{align}
F(\mu, z) 
= & -\frac{2}{\sqrt{s}} + \left(\nu \frac{\partial}{\partial \nu} + 1 - \nu\right)
\int \frac{ds}{\sqrt{s}}\frac{1}{1+\nu s}\\
= & -\frac{2}{\sqrt{s}} + \left(\nu \frac{\partial}{\partial \nu} + 1 - \nu\right)
\left[\frac{2}{\sqrt{\nu}}\tan^{-1}\sqrt{\nu s}\right]\\
= & \left(\frac{s}{1+\nu s} - 2\right)\frac{1}{\sqrt{s}} + \frac{1-2\nu}{\sqrt{\nu}}\tan^{-1}\sqrt{\nu s}\\
= & \frac{3z-2}{\sqrt{z}(1-z)}\sqrt{1-\mu z} + \frac{3-2\mu}{\sqrt{\mu - 1}} 
\tan^{-1}\sqrt{\frac{(\mu - 1)z}{1-\mu z}}
\end{align}
$$
This leads to
$$\begin{align}
& \sqrt{\beta(1-\beta)}\left(
F(\frac{1}{1-\beta},\alpha) - F(\frac{1}{\beta},\alpha)\right)\\
= & \frac{3\alpha-2}{\sqrt{\alpha}(1-\alpha)}
\left(\sqrt{\beta(1-\beta-\alpha)} - \sqrt{(1-\beta)(\beta-\alpha)}\right)\\
&+(1-3\beta)\tan^{-1}\sqrt{\frac{\beta \alpha}{1-\beta-\alpha}}
-(3\beta-2)\tan^{-1}\sqrt{\frac{(1-\beta)\alpha}{\beta-\alpha}}
\end{align}$$
and as a result,
$$\begin{align}
\varphi(\alpha,\beta) 
= &
2\left\{
\frac{3\alpha-2}{\sqrt{\alpha}(1-\alpha)}\left(
 \frac{2\beta-1+\alpha}{\sqrt{\beta(1-\beta-\alpha)}}
-\frac{2\beta-1-\alpha}{\sqrt{(1-\beta)(\beta-\alpha)}}
\right)
+6\left(
\tan^{-1}\sqrt{\frac{\beta \alpha}{1-\beta -\alpha}}
+\tan^{-1}\sqrt{\frac{(1-\beta)\alpha}{\beta-\alpha}}
\right)
-\sqrt{\alpha}
\left(
  \frac{1-3\beta}{(1-\beta)\sqrt{\beta(1-\beta-\alpha)}}
+ \frac{3\beta-2}{\beta\sqrt{(1-\beta)(\beta-\alpha)}}
\right)
\right\}\\
\end{align}$$
A: Here is something to start with. It's just direct computation without any special transformation of the integral, and it only goes so far.
$$\int \frac{1}{((A+t)(B-t))^{3/2}}\,dt=\frac{2(A-B+2t)}{(A+B)^2\sqrt{(A+t)(B-t)}}+C$$
So 
$$
\begin{align*}
&\int_{0}^{\alpha} \int_{0}^{\alpha-x} \frac{1}{[(x+y)(\beta-x)(1-\beta-y)]^{3/2}}\,dy\,dx\\
&=\int_{0}^{\alpha} \frac{1}{(\beta-x)^{3/2}}\left[ \frac{2(x-(1-\beta)+2y)}{(x+1-\beta)^2\sqrt{(x+y)(1-\beta-y)}} \right]_{0}^{\alpha-x}\,dx\\
&=\int_{0}^{\alpha} \frac{2}{(\beta-x)^{3/2}(x+1-\beta)^2}\left[ \frac{-1+\beta+2\alpha-x}{\sqrt{\alpha(1-\beta-\alpha+x)}} -\frac{x-1+\beta}{\sqrt{x(1-\beta)}} \right]\,dx\\
&=\int_{0}^{\alpha} \frac{2}{(\beta-x)^{3/2}(x+1-\beta)^2}\left[ -\frac{1}{\sqrt{\alpha}}\sqrt{1-\beta-\alpha+x} +\frac{\sqrt{\alpha}}{\sqrt{1-\beta-\alpha+x}} -\sqrt{\frac{x}{(1-\beta)}}+\sqrt{\frac{1-\beta}{x}} \right]\,dx
\end{align*}$$
These four integrals may (or may not) prove to be more tractable.
A: Concerning the first integral (with respect to "y"), I obtained a real expression after having introduced a series of assumptions   
Integrate[1/((x + y) (b - x) (1 - b - y))^(3/2), {y, 0, a - x}, 
 Assumptions -> { {a Real}, {a > 0}, {a < 1/2}, {b Real}, {b > 
     a}, {b < 1 - a}, {x Real} , {x > 0}, {x < a}}]  
The result is a ConditionalExpression which expresses several conditions to be fullfilled. May be you could check if you can continue filling the list of assumptions.
