Is it true that in insertion of $n$ GM between $a$ and $b$, the common ratio is positive? On the topic of insertion of geometric mean, the following is written in my book:
Let $G_{1}, G_{2}, \ldots, G_{n}$ be $n$ geometric means between two given numbers $a$ and $b$. Then, $a, G_{1}, G_{2}, \ldots, G_{n}, b$ is a G.P. consisting of $(n+2)$ terms. Let $r$ be the common ratio of this G.P. Then,
$$
\begin{array}{c}
b=(n+2) t h \text { term }=a r^{n+1} \\
\Rightarrow \quad r^{n+1}=\frac{b}{a} \Rightarrow r=\left(\frac{b}{a}\right)^{\frac{1}{n+1}} \\
\Rightarrow \quad G_{1}=a r=a\left(\frac{b}{a}\right)^{1 /(n+1)}, G_{2}=a r^{2}=a\left(\frac{b}{a}\right)^{2 /(n+1)} \ldots, G_{n}=a r^{n}=a\left(\frac{b}{a}\right)^{n / n+1} .
\end{array}
$$
Now I think the author did a mistake in assuming $r$ to be positive. How can he assume it to be positive if he is deriving the formula for the general case?
To give a concrete example suppose we have to insert 5 geometric means between $576$ and $9.$
This is an example from the book, the answer given by the author is as follows:
Let $G_{1}, G_{2}, G_{3}, G_{4}, G_{5}$ be 5 geometric means between $a=576$ and $b=9$. $576, G_{1}, G_{2}, G_{3}, G_{4}, G_{5}, 9$ is a G.P. with common ratio $r$ given by $
\begin{array}{l}
r=\left(\frac{9}{576}\right)^{\frac{1}{5+1}}=\left(\frac{1}{64}\right)^{\frac{1}{6}}=\frac{1}{2} . \quad\left[\text { Using: } r=\left(\frac{b}{a}\right)^{\frac{1}{n+1}}\right]\\
: \quad G_{1}=a r=576 \times \frac{1}{2}=288, \quad G_{2}=a r^{2}=576 \times \frac{1}{4}=144 \text { , }\\
G_{3}=a r^{3}=576 \times \frac{1}{8}=72, \quad G_{4}=a r^{4}=576 \times \frac{1}{16}=36 \text { and }, G_{5}=a r^{5}=576 \times \frac{1}{32}=18\\
\text { Hence, } 288,144,72,36,18 \text { are the required geometric means between } 576 \text { and } 9 \text { . }
\end{array}
$
Again he assumed $r$ to be positive, he didn't at all consider that $r$ could also be $\frac{-1}{2}$. Why is he doing that?
The formula is also derived in a similar way on this and this sites.
 A: $\textbf{Case 1: If $n$ is odd}$
$a,G_1,G_2,...,G_n,b$ are in G.P $\implies$ $a$ and $b$ are of same sign.
$\because \text{common ratio},  r=\left(\frac{b}{a}\right)^{\frac{1}{n+1}}$, $r$ (real, principal root) will always be positive since $\frac{b}{a}>0$. If you take the negative real value of $r$ (which always exists since the root index $n+1$ is even) then the G.M.s can't lie $\textbf{between}$ the positive numbers $a$ and $b$, though they would still form a G.P. Here, you have a choice of choosing the positive real value of $r$ but in Case $\text{2B}$ you don't.
Your specific example of 5 geometric means is covered in this case.
$\textbf{Case 2: If $n$ is even}$
Case $\text{2A}$:  if $a$ and $b$ are of same sign, $r$ will always be positive.
Case $\text{2B}$: if $a$ and $b$ are of opposite sign, $r$ can not be positive.
$\because r=\left(\frac{b}{a}\right)^{\frac{1}{n+1}}$ and $\frac{b}{a}<0$, it is not necessary that a real value of $r$ always exist.
For instance, suppose we want to insert two G.Ms between $ -3$ and $8$. Let $-3,-3r,-3r^2,8$ are in G.P. then the common ratio, $r=(-\frac{8}{3})^\frac{1}{3}$. The principal root here is imaginary. Taking the real root of three 3rd roots of $(-\frac{8}{3})$, we get $r=-1.3867$ (of course it needs to be negative). Fortunately we got a real value of $r$ here, but still our work is not done. Although the definition says,  "if $a,G_1,G_2,...,G_n,b$ are in G.P. then, $G_1,G_2,...,G_n$ are $n$ G.Ms $\textbf{between}$ $a$ and $b$". But for these G.M.s to actually lie between $a$ and $b$, being in such G.P. is not sufficient unless we make some assumption.
Now, the two G.M.s $-3r,-3r^2$ are $4.1601$ and $-5.7688$ all of which don't lie between $ -3$ and $8$. Would you consider them as G.M.s as they form a G.P or do they need to lie between $-3$ and $8$ to be called as two G.M.s $\textbf{between}$ $-3$ and $8$?
Such questions don't actually exist because in the expression for common ratio,  $r=\left(\frac{b}{a}\right)^{\frac{1}{n+1}}$, a tacit assumption is always there that $\frac{b}{a}>0$ (base of an exponential function) and $r$ is the positive principal root. With these assumptions, "if $a,G_1,G_2,...,G_n,b$ are in G.P. then, $G_1,G_2,...,G_n$ are $n$ G.Ms and they always lie between $a$ and $b$ whether both $a$ and $b$ are positive or both are negative."
