$a^2-2ab+a^2b-2b^2=$
$=-2b^2+a(a-2)b+a^2.$
In order to factorize the last polynomial, we have to solve the following quadratic equation:
$-2b^2+a(a-2)b+a^2=0$
that is equivalent to
$2b^2-a(a-2)b-a^2=0.$
$\Delta=a^2(a-2)^2+8a^2=a^2(a^2-4a+12)\ge0,$
$b=\frac{a(a-2)\pm\sqrt{a^2(a^2-4a+12)}}{4}=\frac{a(a-2)\pm a\sqrt{a^2-4a+12}}{4}.$
Now we can factorize the polynomial:
$-2b^2+a(a-2)b+a^2=$
$=-2\left(b-\frac{a(a-2)-a\sqrt{a^2-4a+12}}{4}\right)\cdot\left(b-\frac{a(a-2)+a\sqrt{a^2-4a+12}}{4}\right)=$
$\begin{align*}
&=-\frac{1}{8}\left(4b-a^2+2a+a\sqrt{a^2-4a+12}\right)\cdot\left(4b-a^2+2a-a\sqrt{a^2-4a+12}\right).\\
\end{align*}$
Therefore we get that
$\begin{align}
&a^2-2ab+a^2b-2b^2=\\
&=-\frac{1}{8}\left(4b-a^2+2a+a\sqrt{a^2-4a+12}\right)\cdot\left(4b-a^2+2a-a\sqrt{a^2-4a+12}\right).\\
\end{align}$