In a triangle $ABC$, $AB = a-b$ and $BC = 2\sqrt{ab}$, then find $\angle B$? Is this question solvable?

In $\Delta ABC$, $AB = a-b$ and $BC = 2\sqrt{ab}$, then $\angle B$ is
(a) $\: 60^{\circ}$
(b) $\: 30^{\circ}$
(c) $\: 90^{\circ}$
(d) $\: 45^{\circ}$

 A: HINT:
What do we know from $\triangle\text{ABC}$:


*

*$$\angle\text{A}+\angle\text{B}+\angle\text{C}=180^\circ=\pi\space\text{radians}$$

*For the length $\text{AB}$:
$$\text{AB}=\text{a}-\text{b}$$

*For the length $\text{BC}$:
$$\text{BC}=2\cdot\sqrt{\text{a}\cdot\text{b}}$$


Using the law of sinus and the law of cosines:


*

*$$\frac{\text{BC}}{\sin\left(\angle\text{A}\right)}=\frac{\text{AC}}{\sin\left(\angle\text{B}\right)}=\frac{\text{AB}}{\sin\left(\angle\text{C}\right)}$$

*$$
\begin{cases}
\text{BC}^2=\text{AC}^2+\text{AB}^2-2\cdot\text{AC}\cdot\text{AB}\cdot\cos\left(\angle\text{A}\right)\\
\text{AC}^2=\text{BC}^2+\text{AB}^2-2\cdot\text{BC}\cdot\text{AB}\cdot\cos\left(\angle\text{B}\right)\\
\text{AB}^2=\text{BC}^2+\text{AC}^2-2\cdot\text{BC}\cdot\text{AC}\cdot\cos\left(\angle\text{C}\right)
\end{cases}
$$


So, using your information (and $\left(2\cdot\sqrt{\text{a}\cdot\text{b}}\right)^2=4\cdot\text{a}\cdot\text{b}$):


*

*$$\frac{2\cdot\sqrt{\text{a}\cdot\text{b}}}{\sin\left(\angle\text{A}\right)}=\frac{\text{AC}}{\sin\left(\angle\text{B}\right)}=\frac{\text{a}-\text{b}}{\sin\left(\angle\text{C}\right)}$$

*$$
\begin{cases}
4\cdot\text{a}\cdot\text{b}=\text{AC}^2+\left(\text{a}-\text{b}\right)^2-2\cdot\text{AC}\cdot\left(\text{a}-\text{b}\right)\cdot\cos\left(\angle\text{A}\right)\\
\text{AC}^2=4\cdot\text{a}\cdot\text{b}+\left(\text{a}-\text{b}\right)^2-2\cdot\left(2\cdot\sqrt{\text{a}\cdot\text{b}}\right)\cdot\left(\text{a}-\text{b}\right)\cdot\cos\left(\angle\text{B}\right)\\
\left(\text{a}-\text{b}\right)^2=4\cdot\text{a}\cdot\text{b}+\text{AC}^2-2\cdot\left(2\cdot\sqrt{\text{a}\cdot\text{b}}\right)\cdot\text{AC}\cdot\cos\left(\angle\text{C}\right)
\end{cases}
$$

A: Setting $c=AB=a-b$ gives a degenerate triangle. If there is no relationship between the sides and $a$,$b$,$c$ others than stated any solution is possible. There is a lack of information.
A: Using @JanEerland ‘s suggestion, $BC = k \sin A$, $AC = k \sin B$, and $AB = k \sin C$; for some $k \ne 0$.
Substituting in the cosine law (wrt B), we get
$$\sin^2 B = \sin^2 A + \sin^2 C – 2 \sin A \sin C \cos B$$
(There might be others) but one solution of it is $B = 90^0$ and $A$ is then complement to $C$.
Note that, up to this point, the given $AB = a - b$ and $BC = ...$ have not been used. They probably are used to find $AC = … = a + b$ by Pythagoras theorem, and for the checking of triangle inequality (which maybe or maybe not necessary). This is not difficult to do with the help of $a – b \gt 0$.
However, the notation used in the question is very misleading. This is because according to our common naming convention, for $\triangle ABC$, $BC = a$, $AC = b$ etc. Such confusion can be found in various comments. 
