The correct steps are those given in $(1)$
$$\sqrt[10]{a^5b^2}=\sqrt[10]{a^5}\cdot \sqrt[10]{b^2}=\sqrt a \cdot \sqrt[5]{|b|}$$
instead of the $(2)$ for the following reasons.
Remembering that $a,b\in\Bbb R$ we consider $\sqrt[10]{a^5b^2}$. Since we have a root with an even index $10$ it necessarily needs to be $a\geq 0$. For $b$ into the $\sqrt[10]{\vphantom{b}\ }$, $b$ being squared we not have any problem. This fact it is important to be able to apply distributional property of $n$th root extraction respect to the multiplication. We have
$$ 0\leq \sqrt[10]{a^5b^2}=\sqrt[10]{a^5}\cdot \sqrt[10]{b^2}$$
Being $a\geq 0$ then $\sqrt[10]{a^5}=\sqrt{a}$ and no absolute value is required. For $b\in\Bbb R$ we have
$$\sqrt[10]{b^2}\geq 0$$
and must be $$\sqrt[5]{|b|}\ge 0$$ necessarily with absolute value in order to have concordance of the sign of the two radicals $\sqrt[10]{b^2}$ and $\sqrt[5]{|b|}$, where the first radical is positive or null. If I did not put the absolute value at $b$, and wrote $\sqrt[5]{b}$, this radical can be positive, negative or null and we would have the contradiction
$$0\leq \sqrt[10]{b^2}=\sqrt[5]{b}<0$$
Another example can be $$\sqrt[6]{(x-1)^2}$$ where the existence field is defined for each $\Bbb R$, Being $\sqrt[6]{(x-1)^2}\geq 0$ then by applying the invariant property we must $\sqrt[6]{(x-1)^2}=\sqrt[3]{|x-1|}\geq 0$, absolute value is necessary because the concordance of the initial sign must be preserved.
The invariantive property like other properties of radicals can be applied to simplify radicals if the base of the radical is positive or null; if it is negative, sign concordance could be lost. For example $\sqrt[10]{(-2)^6}\neq \sqrt[5]{(-2)^3}$, in fact the first radical is positive while the second is negative. Instead $\sqrt[9]{(-2)^3}=\sqrt[3]{-2} $ because in this case the concordance of the sign is preserved, in fact although the base is negative, the exponent remains odd, preserving the sign of the base. If the radical has a negative base and in the simplification its exponent changes from even to odd, it is necessary to put the absolute value. Example $\sqrt[10]{(-2)^6}=\sqrt[5]{|-2^3|}\geq 0$.
If the radical is literal the same procedure is followed: whenever studying the sign of the radical we find that the base may be negative, if the exponent of the radical changes from even to odd, we put the modulus to ensure concordance of the sign $\sqrt[10]{x^6}=\sqrt[5]{|x|^3}\geq 0$.
Thank you very much for the comments of the users.