Reasoning behind the trigonometric substitution for $\sqrt{\frac{x-\alpha}{\beta-x}}$ and $\sqrt{(x-\alpha)(\beta-x)}$ In my book, under the topic "Evaluation of Integrals by using Trigonometric Substitutions", it is given, in order to simplify integrals containing the expressions $\sqrt{\frac{x-\alpha}{\beta-x}}$ and $\sqrt{(x-\alpha)(\beta-x)}$, the substitution $x=\alpha\cos^2\theta+\beta\sin^2\theta$ must be used. If I remember this form and the substitution, then it definitely helps simplify the integrand. The first expression gets simplified to $\tan\theta$ and the second to $\sin\theta\cos\theta(\alpha-\beta)$. 
I understand that if we do this kind of substitution we greatly simplify the expression. But how do we determine what to substitute in the first place or in other words, if I forget the substitution, is there any way to determine which substitution works well to simplify the integrand? How did the author figure out this substitution is the best fit for this kind of expression? Was it a guess or is there any mathematical reasoning behind it?
 A: In the first place, one notices that the expression can be "normalized" by means of a linear transform that maps $\alpha,\beta$ to $-1,1$, giving the expressions
$$\sqrt{1-x^2}\text{ and }\sqrt{\frac{1+x}{1-x}}=\frac{1+x}{\sqrt{1-x^2}}.$$
Then the substitution $x=\cos\theta$ comes naturally. We could stop here.

Coming back to the unscaled originals, we have
$$x=\frac{\alpha+\beta+(\beta-\alpha)\cos\theta}2$$
which is also
$$x=\frac{(\alpha+\beta)(\cos^2\frac\theta2+\sin^2\frac\theta2)+(\beta-\alpha)(\cos^2\frac\theta2-\sin^2\frac\theta2)}2=\alpha\sin^2\frac\theta2+\beta\cos^2\frac\theta2.$$
A: The following is intended as an elaboration of Yves Daoust's excellent answer. Given the form $$\sqrt{\frac{x-\alpha}{\beta-x}}$$ to be integrated, let's see what happens under the substitution $x=au+b$ (this is what is meant by a linear transform). We get the expression $$\sqrt{\frac{au+(b-\alpha)}{-au+(\beta-b)}}=\sqrt{\frac{u+(b-\alpha)/a}{-u+(\beta-b)/a}}.$$
In particular, if we choose the constants $a,b$ such that $b-\alpha=a$ and $\beta-b=a$, this transforms the integrand into $$\sqrt{\frac{1+u}{1-u}}=\frac{1+u}{\sqrt{1-u^2}}.$$ At this point, it becomes natural to substitute $u=\cos\theta$, since the Pythagorean identity $\sin^2\theta+\cos^2\theta=1$ now implies that the denominator $\sqrt{1-u^2}$ simply becomes $\sin\theta$, which simplifies the integrand greatly. Similarly, substituting $x=au+b$ into the integrand of the form $\sqrt{(x-\alpha)(\beta-x)}$ provides us with $$\sqrt{(-b^2+(\alpha+\beta)b-\alpha\beta)+u(2ab+a\alpha+a\beta)-a^2u^2}$$
which is the same as
$$ a\sqrt{(-b^2+(\alpha+\beta)b-\alpha\beta)/a^2+u(2b+\alpha+\beta)/a-u^2}. $$
In substance this is just $\sqrt{A+Bu-u^2}$ for constants $A,B$, which after completing the square further becomes something that looks like $\sqrt{1-v^2}$ for an appropriate choice of $v$. Again, this makes the substitution $v=\cos\theta$ natural.
In both cases, if you work carefully through each step in the calculation you will find that it all amounts to the substitution $$x=\frac{\alpha+\beta+(\beta-\alpha)\cos\theta}{2},$$
which as Yves shows can also be written as
$$x=\alpha\sin^2\frac\theta2+\beta\cos^2\frac\theta2.$$
(Note that the fact that the argument of the functions is off by a factor of $2$ from your substitution should not bother you, since it is just a constant and does not affect the efficiency of the substitution.)
A: First, your question is commendable. I also believe it is better to remember the rationale behind a substitution than to memorise mindlessly.
Secondly, you say that the substitution blah-blah must be used. I would have said instead may be used -- there may be some other substitution that one could use, that is.
Now, about your main question. The main thing to note about these expressions is that they involve the square root of a negative quadratic expression, that is, something of the form $\sqrt{-ax^2+bx+c},$ with $a>0.$ For the quotient, see this by rationalising either the numerator or the denominator. Well, one may always deal instead with the case with $a=1$ since otherwise one can factor out the modulus of the leading coefficient, still staying in the real domain.
Once you observe that we're dealing with an integral involving $$\sqrt{c+bx-x^2},$$ then the usual sine substitution should jump to mind. Well, to see this, complete squares to get $$\sqrt{A-\left(x-\frac b2\right)^2},$$ where $A=\frac{b^2}{4}+c.$ Necessarily, we must take only nonnegative values of $A$ to stay in the real domain. Then it's easy to see that this is easily simplified by the sine substitution. To see this even more obviously, factor out the $A$ to get some constant multiple of $$\sqrt{1-y^2},$$ with $$y=\left(\frac{x}{\sqrt A}-\frac{b}{2\sqrt A}\right).$$
Then this is crystal clear.
