Let $x_0 = a$ and $x_{n+1} = b$. I'll assume you are trying to solve following system of equations:
$$x_{k} = \sqrt{x_{k+1}x_{k-1}}\quad\text{ for }\quad 1 \le k \le n$$
Introduce $y_k = \log x_k$, this is equivalent to
$$y_{k+1} - 2y_k + y_{k-1} = 0\quad\text{ for }\quad 1 \le k \le n$$
This is a linear recurrence relation with characteristic polynomial $\lambda^2 - 2\lambda + 1 = (\lambda-1)^2$.
Since the characteristic polynomial has a double root at $\lambda = 1$,
the general solution of it has the form $y_k = A + Bk$ for suitably chosen constants $A, B$.
Since $y_0 = \log a$ and $y_{n+1} = \log b$, we find
$\displaystyle\;A = \log a, B = \frac{\log b - \log a}{n+1}$.
As a result, for any $1 \le k \le n$, we have $$y_k = \log a + \frac{\log b - \log a}{n+1} k \quad\iff\quad
x_k = a \left(\frac{b}{a}\right)^{\frac{k}{n+1}}$$