Locating possible complex solutions to an equation involving a square root Note after reviewing answers. This question illustrates in a non-trivial way how the choice of how to compute the square root in the complex plane can make real difference to the answer you  get.
In this question finding all the solutions came down to solving:
$$ab-1=\sqrt{a^2+b^2+1}$$ subject to $$(a+1)(b+1)=1$$
The condition implies $a^2+b^2+1=(ab-1)^2$, but this leaves open the possibility of $\sqrt {a^2+b^2+1}=1-ab$ rather than $ab-1$.
The solution and discussion on the original question was getting rather long, and dealt mainly with real roots, so I've extracted this sub-question on complex roots.
Could someone please explain what constraints can be put on the complex solutions of this equation - the values of $a$ and $b$ - which would be consistent with the square root operation. It would help if answers were elementary and didn't assume a prior knowledge of complex square roots.
 A: Every nonzero complex number $z$ has two "square roots".  You might call one $\sqrt{z}$ and the other $-\sqrt{z}$, but which is which?  A particular choice (for all $z$ in some region of the complex plane) is called a branch of the function $\sqrt{z}$.  There is no single branch that is continuous everywhere: if you start at some point with a particular choice of $\sqrt{z}$ and travel in a loop around $0$, keeping $\sqrt{z}$ continuous along your path, when you come back to the start $\sqrt{z}$ will have $-$ its original value.  A curve $\Gamma$ in whose complement your branch is continuous is called a branch cut.
One popular choice, called the "principal branch", is to make the real part of $\sqrt{z}$ always $\ge 0$.  The branch cut in this case is the negative real axis.
A: Let's pursue Robert Israel's line of thinking, and take $a$ as the complex number $x+iy$, with $b=-\frac a{a+1}$.
We define the principal value of the square root of a complex number as having non-negative real part, with the square root of a negative real number being a value on the positive imaginary axis (when the real part is zero)
Then the condition for $(a,b)$ to be a solution is that $ab-1$ has positive real part, or lies on the positive imaginary axis. This guarantees that it is in the half-plane which contains the square root we are looking for. The alternative possibility lies in the other half plane.
Now $$ab-1=-\frac{a^2}{a+1}-1=-\frac {a^2+a+1}{a+1}=-a-\frac 1{a+1}$$$$=-x-iy-\frac1{(x+1)+iy}=-x-iy-\frac{x+1-iy}{(x+1)^2+y^2}$$
Now take the real part $$-\frac {x((x+1)^2+y^2)+(x+1)}{(x+1)^2+y^2}$$ We are interested only in the sign, and the denominator is always positive, so our condition becomes $$-(x((x+1)^2+y^2)+(x+1))\gt 0$$ or removing the minus sign $$x((x+1)^2+y^2)+(x+1)\lt 0$$.
Clearly this implies $x\lt 0$ (otherwise both terms are non-negative, and there is no case of equality). If $x\lt -1$ both terms on the left-hand side are negative, and the equation is satisfied whatever $y$ is.
If $x=-1$ then the only condition is easily seen to be that $y$ is non-zero. If $x=-1, y=0$ the basic condition is not satisfied, so there is no solution.
In the case $-1\lt x \lt 0$ we note that $x+1\gt 0$ and $$xy^2\lt-x(x+1)^2-(x+1)$$ $x$ is negative, so when we divide by $x$ we reverse the inequality$$y^2\gt-(x+1)^2-1-\frac 1x$$
A: An alternative choice for the square root is in the upper half plane - with positive imaginary part, and including the non-negative real line where the imaginary part is zero. (see my other answer where I took Robert Israel's suggestion of the right half plane, positive real part). This answer is to illustrate that the choice matters.
We still have$$ab-1=-\frac{a^2}{a+1}-1=-\frac {a^2+a+1}{a+1}=-a-\frac 1{a+1}$$$$=-x-iy-\frac1{(x+1)+iy}=-x-iy-\frac{x+1-iy}{(x+1)^2+y^2}$$
But now we take the imaginary part $$-y-\frac {y}{(x+1)^2+y^2}=-\frac{y(x+1)^2+y^3+y}{(x+1)^2+y^2}$$
Again the denominator is positive, so we want to control the sign of the numerator by ensuring $$y((x+1)^2+y^2+1)\lt 0$$
This is $y$ times a term which is easily seen to be positive, so the condition is $y\lt0$
The case with $y=0$ is the real case, which is excluded in the question because it has been analysed separately.
Note that this choice of  branch cut (along the positive real axis) leads to a much simpler condition than the more standard alternative. So this is a nice illustration of how even simple definitions can make a significant difference.
A: a²+b²+1 = (ab-1)²,
a²+b²+1=a²b²-2ab+1,
(a+b)²=a²b²,
So a+b = ab or a+b = -ab ,
Second constraint.
(a+1)(b+1)=1, 
ab+a+b=0
eliminate a+b gives either 0=0 (true statement all a,b) or 2ab=0 implying a = 0 or b = 0
From your original problem, is their a possibility perhaps that either a = 0 or b = 0 ???
A: (A solution using too much algebra.)
I show that
the solutions are
$a=0, b=0$
and
$a=1, b=-1/2$
and,
in both of these,
the minus sign has to be taken
in the $\sqrt{}$.
If
$(a+1)(b+1)=1$,
then
$b = \dfrac1{a+1}-1$
so
$\begin{align}
ab-1
&=a(\frac1{a+1}-1)-1\\
&=a\frac1{a+1}-(a+1)\\
&=\frac{a-(a+1)^2}{a+1}\\
&=\frac{-a^2-a-1}{a+1}\\
&=-\frac{a^2+a+1}{a+1}\\
\end{align}
$
and
$\begin{align}
a^2+b^2+1
&=a^2+(\frac1{a+1}-1)^2+1\\
&=a^2+(\frac1{a+1})^2-2\frac1{a+1}+2\\
&=\frac{a^2(a+1)^2+1-2(a+1)+2(a+1)^2}{(a+1)^2}\\
&=\frac{(a^2+2)(a+1)^2+1-2(a+1)}{(a+1)^2}\\
&=\frac{(a^2+2)(a^2+2a+1)-2a-1}{(a+1)^2}\\
&=\frac{a^4+2a^3+a+2a^2+4a+2-2a-1}{(a+1)^2}\\
&=\frac{a^4+2a^3+2a^2+3a+1}{(a+1)^2}\\
\end{align}
$
Putting these in 
$ab-1=\sqrt{a^2+b^2+1}
$,
$-\dfrac{a^2+a+1}{a+1}
=\sqrt{\dfrac{a^4+2a^3+2a^2+3a+1}{(a+1)^2}}
$.
Multiplying both sides by
$a+1$,
$-(a^2+a+1)
=\sqrt{a^4+2a^3+2a^2+3a+1}
$.
Since
$(a^2+a+1)^2
=a^4+2a^3+3a^2+2a+1
$,
squaring and subtracting gives
$3a^2+2a = 2a^2+3a$
or
$a^2 = a$,
so
$a=0$
or $a=1$.
If $a=0$,
then $b=0$
and we have to take
the negative sign in the $\sqrt{}$.
If $a=1$,
then $b=-1/2$,
$ab-1 = -3/2$,
and
$a^2+b^2+1
=1+1/4+1
=9/4
= (3/2)^2
$,
and,
again,
we have to take
the negative sign in the $\sqrt{}$.
