Differentiate $|x^5|$.
I know the formula for the derivative of absolute value but I can't seem to apply it to get $5x|x^3|$.
$$|x|=\sqrt{x^2},$$
therefore $|x^5|=\sqrt{x^{10}}.$
Then $$|x^5|'=\frac{10x^9}{2\sqrt{x^{10}}}=\frac{5x^9}{|x|^5}=5\frac{x^9}{x^4|x|}=5\frac{x^5}{|x|}=5\frac{|x|^4x}{|x|}=5x|x^3|$$ as desired.
The derivative of $t\mapsto |t|$ can be expressed by $\dfrac{|t|}{t}=\dfrac{t}{|t|}$ (for $t\ne0$); so, with the chain rule, if $f(x)=|x|^5$ you have $$ f'(x)=5|x|^4\frac{x}{|x|}=5x|x|^3 $$ for $x\ne0$. At $0$ the derivative is $0$, because the function $f$ is continuous and $\lim_{x\to0}f'(x)=0$, so the expression $f'(x)=5x|x|^3=5x^3|x|$ is valid for all $x$.
Note: $x^5$ can be simplified to $x^4\cdot|x|$
Apply the product rule: $\frac{\text{d}}{\text{d}x}(x^4\cdot|x|)$:
$$\frac{\text{d}}{\text{d}x}(x^4\cdot|x|)=$$
$$\frac{\text{d}}{\text{d}x}(x^4)\cdot|x|+x^4\cdot\frac{\text{d}}{\text{d}x}(|x|)=$$
$$4x^3\cdot|x|+x^4\cdot\frac{x}{|x|}=$$
$$4x^3\cdot|x|+\frac{x^5}{|x|}=$$
Since $|x^2|=x^2$
$$4x^3\cdot|x|+\frac{x^3\cdot|x^2|}{|x|}=$$
$$4x^3\cdot|x|+x^3\cdot|x|=$$
$$5x^3\cdot|x|$$