How to calculate this hard integral $\int_0^{\infty} \frac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx$? How to prove that $\displaystyle \int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx= \frac{1}{4}\pi^2\sqrt{2}-\sqrt{2}\ln^2\left(1+\sqrt{2}\right)$ ?
It's very difficult and I have no idea.
 A: This can be reduced to a class of integral related to polylogarithm, and is related to Legendre-chi function $\chi_2$. The formula
$$\int_{0}^{\frac{\pi}{2}} \arctan(r \sin\theta) \, d\theta
= 2 \chi_{2} \left( \frac{\sqrt{1+r^{2}} - 1}{r} \right) $$
gives $$\tag{1}\int_{0}^{\frac{\pi}{2}} \arctan( \sin\theta) \ d\theta
= 2 \chi_{2} \left( \sqrt{2}-1 \right) = \frac{\pi^2}{8} - \frac{1}{2}\log^2(\sqrt{2}+1)$$
where we used a famous special value of $\chi_2$. For a "quick introduction" about $\chi_2$, see Sangchul Lee's here.

Return to your problem, denote your integral by $I$, note that
$$\int \frac{\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}} dx = \sqrt 2 \arctan \left[ {\frac{{\sqrt {\sqrt {1 + {x^2}}  - 1} }}{{\sqrt 2 }}} \right]$$
so integration by parts  gives
$$I = \frac{\sqrt{2}\pi^2}{4}-\sqrt{2}\int_0^\infty  {\arctan \left[ {\frac{{\sqrt {\sqrt {1 + {x^2}}  - 1} }}{{\sqrt 2 }}} \right]} \frac{{dx}}{{1 + {x^2}}}$$ 
making $x\mapsto \tan x$ into the last integral gives
$$I = \frac{\sqrt{2}\pi^2}{4}-\sqrt{2}\underbrace{\int_0^{\pi/2}  {\arctan \left( {\frac{{\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)dx}}_J $$

Denote $$F(r)=\int_0^{\pi/2}  {\arctan \left( {\frac{{a\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)} dx$$
Then compute $F'(r)$ and do an indefinite integral gives
$$\begin{aligned} F'(r) &= \int_0^{\pi /2} {\frac{{\sqrt {\cos x} \sin (\frac{x}{2})}}{{\cos x + {a^2}\sin {{(\frac{x}{2})}^2}}}} dx \\
& =  - \left[ {\frac{{2a\arctan \left[ {\frac{{a\cos (\frac{x}{2})}}{{\sqrt {1 - {a^2}} \sqrt {\cos x} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2\sqrt 2 }}{{2 - {a^2}}}\ln \left[ {\sqrt 2 \cos (\frac{x}{2}) + \sqrt {\cos x} } \right]} \right]_0^{\pi /2} \\ &=  - \frac{{\pi a}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2a\arctan \left[ {\frac{a}{{\sqrt {1 - {a^2}} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2\sqrt 2 \ln (\sqrt 2  + 1)}}{{2 - {a^2}}}\end{aligned}$$
Then to get $J$, we integrate the whole expression with respect to $a$ from $0$ to $1$, only the second integral deserves attention, applying $a\mapsto \sin x$:
$$\begin{aligned}\int_0^1 {\frac{{\arctan \left[ {\frac{a}{{\sqrt {1 - {a^2}} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }}da}  &= \int_0^{\pi /2} {\frac{{x\cos x}}{{2 - {{\sin }^2}x}}dx}  \\&=- \int_0^{\frac{\pi }{2}} {xd\left[ {\arctan (\cos x)} \right]}  \\&= \int_0^{\frac{\pi }{2}} {\arctan (\cos x)dx} \end{aligned}$$
Then use $(1)$. Combining all above evaluation should give
$$J=\int_0^{\pi/2}  {\arctan \left( {\frac{{\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)}dx=\ln^2(\sqrt{2}+1)$$ this proves your result. 
Probably there is some smart way to reduce $J$ directly into $(1)$, I hope someone can point on how.
A: Let $x = \tan u \implies dx = \sec^2u\ du$
$$\begin{align}\int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx =& \int_0^{\pi/2} \dfrac{\arctan(\tan u)\sqrt{\sec u-1}\sec u}{\tan u }\,du &\\=& \int_0^{\pi/2} \dfrac{u\sqrt{\sec u-1}}{\sin u }\,du&\\=& \int_0^{\pi/2} \dfrac{u\sqrt{1-\cos u}}{\sin u \sqrt{\cos u}}\,du&\\=& \sqrt{2}\int_0^{\pi/2} \dfrac{u\sin u/2}{\sin u \sqrt{\cos u}}\,du &\\=& \dfrac{1}{\sqrt{2}}\int_0^{\pi/2} \dfrac{u}{\cos (u/2) \sqrt{\cos u}}\,du&\\=& \dfrac{1}{\sqrt{2}}\left(\left[2u \sin^{-1}\left(\tan \dfrac u2\right)\right]_0^{\pi/2} - 2\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du\right)&\\=& \dfrac{\pi^2}{2\sqrt{2}} - \dfrac{2}{\sqrt{2}}\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du \end{align}  $$
Let $z = u/2 \implies 2dz = du$
$$\begin{align}\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du&=& 2\int_0^{\pi/4}\sin^{-1}(\tan z )\, dz\end{align}$$
Let $t = \tan z$
$$\begin{align} 2\int_0^{\pi/4}\sin^{-1}(\tan z )\, dz=& 2\int_0^1\dfrac{\sin^{-1} t}{t^2 + 1} dt&\\=&  2[\tan^{-1} x \sin^{-1} x]_0^1 -
 2\int_0^1 \dfrac{\tan^{-1}(t)}{\sqrt{1-t^2}}\, dt&\\=&  2\dfrac{\pi^2}{8} -
 2\int_0^1 \dfrac{\tan^{-1}(t)}{\sqrt{1-t^2}}\, dt\end{align}$$
The integral $$\int_0^1 \dfrac{\tan^{-1}(t)}{\sqrt{1-t^2}}\, dt =  -\frac{\ln^{2}(1+\sqrt{2})}{2} + \frac{\pi^{2}}{8} \tag 1$$
So,
$$\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du = \ln^2(1+ \sqrt{2})$$ 
The final answer is then 
$$\dfrac{\pi^2}{2\sqrt{2}} - \sqrt{2}\ln^2(1+\sqrt{2})$$.
For the proof of $(1)$ see here.
A: $\begin{align}I&=\int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx
\end{align}$
Perform the change of variable $x=\dfrac{2y}{1-y^2}$,
$\begin{align}I&=\sqrt{2}\int_0^{1} \dfrac{\arctan\left(\dfrac{2y}{1-y^2}\right)}{\sqrt{1-y^2}}\,dy\\
&=2\sqrt{2}\int_0^{1} \dfrac{\arctan y}{\sqrt{1-y^2}}\,dy\\
\end{align}$
If one assume that,
$\displaystyle \int_0^{1} \dfrac{\arctan y}{\sqrt{1-y^2}}\,dy=\frac{1}{8}\pi^2-\frac{1}{2}\ln^2(\sqrt{2}+1)$
therefore,
$\displaystyle \boxed{I=\frac{1}{4}\pi^2\sqrt{2}-\sqrt{2}\ln^2(\sqrt{2}+1)}$
(for a proof of this, see: Evaluating a sum involving binomial coefficient in denominator )
A: Through the substitution $x=\tan\theta$ the integral is converted into
$$ \int_{0}^{\pi/2}\frac{\theta}{\sin\theta}\sqrt{\frac{1}{\cos\theta}-1}\,d\theta=\int_{0}^{\pi/2}\frac{\theta}{\cos\frac{\theta}{2}\sqrt{2\cos\theta}}\,d\theta$$
and by integration by parts the RHS equals
$$ \frac{\pi^2}{2\sqrt{2}}-\sqrt{2}\int_{0}^{\pi/2}\arctan\left(\frac{\sin\frac{\theta}{2}}{\sqrt{\cos\theta}}\right)\,d\theta $$
where
$$\begin{eqnarray*}\int_{0}^{\pi/2}\arctan\left(\frac{\sin\frac{\theta}{2}}{\sqrt{\cos\theta}}\right)\,d\theta &=& \int_{0}^{\pi/2}\arctan\sqrt{\frac{1-\cos\theta}{2\cos\theta}}\,d\theta\\&=&\int_{0}^{\pi/2}\arctan\sqrt{\frac{1-\sin\theta}{2\sin\theta}}\,d\theta\\&=&\int_{0}^{1}\arctan\sqrt{\frac{1-x}{2x}}\,\frac{dx}{\sqrt{1-x^2}}\\&=&\int_{0}^{+\infty}\frac{\arctan\sqrt{t}}{(1+2t)\sqrt{t(1+t)}}\,dt\\&=&\int_{0}^{+\infty}\frac{2\arctan u}{(1+2u^2)\sqrt{1+u^2}}\,du\end{eqnarray*}$$
by Feynman's trick, is related to the integral
$$ \int_{0}^{1}\frac{K(\sqrt{x})}{\sqrt{2-x}}\,dx = \frac{\pi^2}{4}-\log^2(1+\sqrt{2}),$$
which can be computed through Fourier-Legendre series expansion (see here, page 19) and the dilogarithm reflection formula.
