Let $I$ denote the integral. Then, using integration by parts we can write
$$I=\int_{0}^{\pi\over 2}\frac{\left(\tan^{-1}(\sin^2 x) \right)^2}{\sin^2 x}dx = 4\int_0^{\frac{\pi}{2}}\frac{\cos^2 x\tan^{-1}(\sin^2 x)}{1+\sin^4 x}dx$$
The main idea of this evaluation is to use differentiation under the integral sign. Let us introduce the parameter $\alpha$: $$f(\alpha)=\int_0^{\frac{\pi}{2}}\frac{\cos^2 x\tan^{-1}(\alpha \sin^2 x)}{1+\sin^4 x}dx$$
Taking derivative inside the integral,
\begin{align*}
f'(\alpha) &= \int_0^{\pi\over 2}\frac{\cos^2 x}{1+\sin^4 x}\cdot\frac{\sin^2 x}{1+\alpha^2 \sin^4 x}dx \\
&= \frac{1}{1-\alpha^2}\int_0^{\pi\over 2}\frac{\cos^2 x\sin^2 x}{1+\sin^4 x}dx-\frac{\alpha^2}{1-\alpha^2}\int_0^{\pi\over 2}\frac{\cos^2 x\sin^2 x}{1+\alpha^2 \sin^4 x}dx \quad (\text{Partial Fractions})
\end{align*}
Let $g(\alpha)=\int_0^{\pi\over 2}\frac{\cos^2 x\sin^2 x}{1+\alpha^2 \sin^4 x}dx$. Then,
$$\frac{I}{4}=f(1)=\int_0^1\frac{g(1)-\alpha^2 g(\alpha)}{1-\alpha^2}d\alpha \tag{1}$$
Evaluation of $g(\alpha)$
\begin{align*}
g(\alpha) &= \int_0^{\frac{\pi}{2}}\frac{\cos^2 x\sin^2 x}{1+\alpha^2 \sin^4 x}dx \\
&= \int_0^\infty \frac{t^2}{\left(t^4(1+\alpha^2)+2t^2+1 \right)(t^2+1)}dt \quad (t=\tan x)\\
&= -\frac{1}{\alpha^2}\int_0^\infty\frac{1}{1+t^2}dt+\frac{1}{\alpha^2}\int_0^\infty\frac{1+(1+\alpha^2)t^2}{(1+\alpha^2)t^4+2t^2+1}dt \\
&= -\frac{\pi}{2\alpha^2}+\frac{\pi \sqrt{1+\sqrt{1+\alpha^2}}}{2\sqrt{2}\alpha^2}\tag{2}
\end{align*}
That last integral was evaluated using an application of the residue theorem. Substitute this into equation (1) to get
$$I=\pi\sqrt{2}\int_0^1\frac{\sqrt{1+\sqrt{2}}-\sqrt{1+\sqrt{1+\alpha^2}}}{1- \alpha ^2 }d\alpha \tag{3}$$
Evaluation of integral (3)
Luckily, integral (3) has a nice elementary anti-derivative. \begin{align*}
&\; \int \frac{\sqrt{1+\sqrt{2}}-\sqrt{1+\sqrt{1+\alpha^2}}}{1-\alpha^2}d\alpha \\ &= \sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha) -\int \frac{\sqrt{1+\sqrt{1+\alpha^2}}}{1-\alpha^2} d\alpha \\
&= \sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)-\int \frac{t\sqrt{1+t}}{(2-t^2)\sqrt{t^2-1}}dt\quad (t=\sqrt{1+\alpha^2}) \\
&= \sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)-\int \frac{t}{(2-t^2)\sqrt{t-1}}dt \\
&=\sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)- 2\int \frac{u^2+1}{2-(u^2+1)^2}du\quad (u=\sqrt{t-1}) \\
&=\sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)- 2\int \frac{u^2+1}{(\sqrt{2}-1-u^2)(\sqrt{2}+1+u^2)}du \\
&= \sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)-\int\frac{du}{\sqrt{2}-1-u^2}+\int\frac{du}{\sqrt{2}+1+u^2} \\
&=\sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)-\sqrt{\sqrt{2}+1}\tanh^{-1}\left( u \sqrt{\sqrt{2}+1}\right)+\sqrt{\sqrt{2}-1}\tan^{-1}\left(u\sqrt{\sqrt{2}-1} \right) +C\\
&= \sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)-\sqrt{\sqrt{2}+1}\tanh^{-1}\left( \sqrt{\sqrt{1+\alpha^2}-1} \sqrt{\sqrt{2}+1}\right)\\ &\quad +\sqrt{\sqrt{2} -1}\tan^{-1}\left(\sqrt{\sqrt{1+\alpha^2}-1}\sqrt{\sqrt{2}-1} \right) +C
\end{align*}
Therefore, the integral is
\begin{align*}
I &= \pi \sqrt{2} \lim_{\alpha\to 1}\Bigg\{ \sqrt{1+\sqrt{2}}\tanh^{-1}(\alpha)-\sqrt{\sqrt{2}+1}\tanh^{-1}\left( \sqrt{\sqrt{1+\alpha^2}-1} \sqrt{\sqrt{2}+1}\right)\\ &\quad +\sqrt{\sqrt{2} -1}\tan^{-1}\left(\sqrt{\sqrt{1+\alpha^2}-1}\sqrt{\sqrt{2}-1} \right) \Bigg\} \\
&= \pi\sqrt{2}\left\{\frac{\sqrt{\sqrt{2}+1}}{2}\log\left(\frac{\sqrt{2}+2}{4} \right) +\sqrt{\sqrt{2}-1}\tan^{-1}\left(\sqrt{2}-1 \right)\right\} \\
&= \pi\sqrt{2}\left\{\frac{\sqrt{\sqrt{2}+1}}{2}\log\left(\frac{\sqrt{2}+2}{4} \right) +\frac{\pi}{8}\sqrt{\sqrt{2}-1}\right\} \\
&= \color{maroon}{\pi \log\left( \frac{2+\sqrt{2}}{4}\right) \sqrt{\frac{\sqrt{2}+1}{2}}+\frac{\pi^2}{4}\sqrt{\frac{\sqrt{2}-1}{2}}}\approx 0.576335
\end{align*}