Other answers offer alternative approaches to integrating form the beginning. If you are looking to find where your steps went astray, it starts when you are using: $$2(1-\cos(2a))=4\sin^2(4a)$$ but the correct identity is: $$2(1-\cos(2a))=4\sin^2(a)$$ Just check both sides with $a=\pi/4$ and you'll believe it. And similarly for the next identity that you use: $2(1+\cos(2a))=4\cos^2(4a)$ should be $2(1+\cos(2a))=4\cos^2(a)$.
From there, you would have
$$I=\sin2a\left(\frac1{2\sin a}\arctan\left(\frac{u-u^{-1}}{2\sin(a)}\right)+\frac1{4\cos a}\ln\left|\frac{u+u^{-1}-2\cos a}{u+u^{-1}+2\cos a}\right|\right)+C$$
which gives
$$I=\cos(a)\arctan\left(\frac{u-u^{-1}}{2\sin(a)}\right)+\frac1{2}\sin(a)\ln\left|\frac{u+u^{-1}-2\cos a}{u+u^{-1}+2\cos a}\right|+C$$
And then we start back-substituting. We reach a point where it helps to use $\arctan\left(\frac{A}{B}\right)=\arcsin\left(\frac{A}{\sqrt{A^2+B^2}}\right)=\frac{\pi}{2}-\arccos\left(\frac{A}{\sqrt{A^2+B^2}}\right)$ and the constant can be absorbed into the $C$. Keep in mind that expressions in $a$ are constant for the purposes of $C$. Then we reach a point where by adding a certain logarithmic expression in $a$ (again, absorbed into $C$) we can simplify the appearance of the logarithmic term.
$$
\begin{align}
I&=\cos(a)\arctan\left(\frac{\sqrt\frac{\sin(x-a)}{\sin(x+a)}-\sqrt\frac{\sin(x+a)}{\sin(x-a)}}{2\sin(a)}\right)\\
&\phantom{=}{}+\frac1{2}\sin(a)\ln\left|\frac{\sqrt\frac{\sin(x-a)}{\sin(x+a)}+\sqrt\frac{\sin(x+a)}{\sin(x-a)}-2\cos a}{\sqrt\frac{\sin(x-a)}{\sin(x+a)}+\sqrt\frac{\sin(x+a)}{\sin(x-a)}+2\cos a}\right|+C\\
&=\cos(a)\arctan\left(\frac{\sin(x-a)-\sin(x+a)}{2\sin(a)\sqrt{\sin(x-a)\sin(x+a)}}\right)\\
&\phantom{=}{}+\frac1{2}\sin(a)\ln\left|\frac{\sin(x-a)+\sin(x+a)-2\cos(a)\sqrt{\sin(x-a)\sin(x+a)}}{\sin(x-a)+\sin(x+a)+2\cos(a)\sqrt{\sin(x-a)\sin(x+a)}}\right|+C\\
&=\cos(a)\arctan\left(\frac{-\cos(x)}{\sqrt{\sin(x-a)\sin(x+a)}}\right)\\
&\phantom{=}{}+\frac1{2}\sin(a)\ln\left|\frac{\sin(x)-\sqrt{\sin(x-a)\sin(x+a)}}{\sin(x)+\sqrt{\sin(x-a)\sin(x+a)}}\right|+C\\
&=-\cos(a)\arcsin\left(\frac{\cos(x)}{\sqrt{\cos^2(x)+\sin(x-a)\sin(x+a)}}\right)\\
&\phantom{=}{}+\frac1{2}\sin(a)\ln\left|\frac{\sin(x)-\sqrt{\sin(x-a)\sin(x+a)}}{\sin(x)+\sqrt{\sin(x-a)\sin(x+a)}}\right|+C\\
&=-\cos(a)\left(\frac{\pi}{2}-\arccos\left(\frac{\cos(x)}{\sqrt{\cos^2(x)+\frac12\cos(2a)-\frac12\cos(2x)}}\right)\right)\\
&\phantom{=}{}+\frac1{2}\sin(a)\ln\left|\frac{\sin(x)-\sqrt{\sin(x-a)\sin(x+a)}}{\sin(x)+\sqrt{\sin(x-a)\sin(x+a)}}\right|+C\\
&=\cos(a)\arccos\left(\frac{\cos(x)}{\cos(a)}\right)+\frac1{2}\sin(a)\ln\left|\frac{\sin(x)-\sqrt{\sin(x-a)\sin(x+a)}}{\sin(x)+\sqrt{\sin(x-a)\sin(x+a)}}\right|+C_1
\end{align}$$
Note that $-\cos(a)\frac{\pi}{2}$ has been absorbed into $C$.
It remains to show $$\frac1{2}\ln\left|\frac{\sin(x)-\sqrt{\sin(x-a)\sin(x+a)}}{\sin(x)+\sqrt{\sin(x-a)\sin(x+a)}}\right|=-\ln\left|\sin(x)+\sqrt{\sin^2(x)-\sin^2(a)}\right|+C_2(a)$$
If you add $\ln\left|\sin(x)+\sqrt{\sin^2(x)-\sin^2(a)}\right|$ to the left side, you have $$
\begin{align}
&\ln\left|\sqrt{\sin(x)-\sqrt{\sin(x-a)\sin(x+a)}}\sqrt{\sin(x)+\sqrt{\sin(x-a)\sin(x+a)}}\right|\\
&=\ln\left|\sqrt{\sin^2(x)-\sin(x-a)\sin(x+a)}\right|\\
&=\ln\left|\sqrt{\sin^2(x)-\frac12\cos(2a)+\frac12\cos(2x)}\right|\\
&=\ln\left|\sqrt{\sin^2(a)}\right|=C_2(a)\\
\end{align}$$
which establishes that last relation.
Also this write up has a typo at an earlier line, with $$\csc^2(x+a)=\frac{\cos^2(2a)+u^4-2u^2\cos2(2a)+\sin^2(2a)}{\csc^2(2a)}$$ but you meant $$\csc^2(x+a)=\frac{\cos^2(2a)+u^4-2u^2\cos(2a)+\sin^2(2a)}{\sin^2(2a)}$$ In any case, these things have been corrected at the next line.