Alternatively you can rewrite your number in its polar form:
- $1+2i=\sqrt 5e^{i\theta _1}$, for some real $\theta_1$;
- $(1+\sqrt 3i)=2e^{i\theta _2}$, for some real $\theta_2$;
- $(1+i)^3=(\sqrt 2e^{i\theta _3})^3$, for some real $\theta _3$.
Yielding $\displaystyle \left\vert\dfrac{\left( 1+2i \right)\left( 1+\sqrt{3} i \right)}{\left( 1+i \right)^{3}}\right \vert=\left \vert\dfrac{\sqrt 5e^{i\theta _1}\cdot2^{i\theta _2}}{2\sqrt 2e^{3\theta _3 i}}\right \vert=\sqrt{\dfrac{5}{2}}.$
In your work you get $\dfrac{1}{2(-1+i)}=\dfrac{-1-i}{2}$ which is wrong.
Correct is $\dfrac{1}{2(-1+i)}=\dfrac{-1-i}{4}$.
This still doesn't excuse for the absolute value of what you get being negative, but that becomes unimportant.
In any case you squared $2\sqrt 3-\sqrt 3$ wrong:
$$4\cdot 3-4\cdot 3+3=(2\sqrt 3-\sqrt 3)~2\neq 4\color{red}{\sqrt 3}-4\cdot 3+3.$$