Two proofs will be given. One proof is the remaining of this answer, along lines of Schläfli and Coxeter. Second proof, more direct, is given at this answer's remark.
Note that
$$\int_{\sqrt {3/5} }^1 {\frac{{\arctan x}}{{\sqrt {2{x^2} - 1} (3{x^2} - 1)}}dx} = \int_1^{\sqrt {5/3} } {\frac{{x(\frac{\pi }{2} - \arctan x)}}{{\sqrt {2 - {x^2}} (3 - {x^2})}}dx} = \frac{{{\pi ^2}}}{{24}} - \frac{1}{2}\int_1^{5/3} {\frac{{\arctan \sqrt x }}{{\sqrt {2 - x} (3 - x)}}dx}$$
therefore your integral is equivalent to
$$\int_1^{5/3} {\frac{{\arctan \sqrt x }}{{\sqrt {2 - x} (3 - x)}}dx} = \frac{{11{\pi ^2}}}{{240}}$$
I will prove this by establishing
$$\tag{1}\int_0^1 {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = \frac{{{\pi ^2}}}{{48}}$$
$$\tag{2} \int_0^{5/3} {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = \frac{{{\pi ^2}}}{{15}}$$
The following ideas are an expounded version of Polylogarihm and Associated Functions by Leonard Lewin page 115-117, arguments there are very terse.
Let $$S(\alpha ,\beta ,\gamma ) = \sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{{{n^2}}}(\cos 2n\alpha - \cos 2n\beta + \cos 2n\gamma - 1)} - {\alpha ^2} + {\beta ^2} - {\gamma ^2}$$
where $0\leq \alpha,\gamma \leq \pi/2, 0\leq \beta < \pi$ and $k$ is a function of $\alpha,\beta,\gamma$ which will be determined soon.
It is easy to check that
$$\frac{{\partial S}}{{\partial \alpha }} = - 2\sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{n}\sin 2n\alpha } - 2\alpha = - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)$$
Therefore the differential $dS$ is
$$\begin{aligned}dS &= \frac{1}{k}\sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{n}(\cos 2n\alpha - \cos 2n\beta + \cos 2n\gamma - 1)} dk - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)d\alpha+\\ & 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \beta } \right)d\beta - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \gamma } \right)d\gamma \end{aligned}$$
Now choose $k$ such that the coefficient of $dk$ vanishes, using
$\sum\limits_{n = 1}^\infty {\frac{{{a^n}\cos nx}}{n}} = - \frac{1}{2}\ln ({a^2} - 2a\cos x + 1)$
one easily see such $k$ is
$$ k = \frac{{\sqrt {{{\cos }^2}\alpha {{\cos }^2}\gamma - {{\cos }^2}\beta } - \sin \alpha \sin \gamma }}{{\sqrt {{{\cos }^2}\alpha {{\cos }^2}\gamma - {{\cos }^2}\beta } + \sin \alpha \sin \gamma }}$$
This completes the definition of $S(\alpha,\beta,\gamma)$. Note that in order for $k$ to be real, we need to assume the term inside radical is always $\geq 0$, we confine ourselves exclusive to this case. Now $dS$ becomes
$$\tag{3}dS = - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)d\alpha+ 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \beta } \right)d\beta - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \gamma } \right)d\gamma $$
Four observations:
- $S(0,\beta,\gamma) = \pi(\beta-\gamma)$
- When $\sin^2 \alpha + \sin^2 \gamma = \sin^2 \beta$, $S(\alpha,\beta,\gamma) = -\alpha^2+\beta^2-\gamma^2$
- When $\cos \alpha \cos\gamma = \cos\beta$, $S(\alpha,\beta,\gamma)= 0$.
- $S(\alpha,\pi - 2\alpha,\alpha) = 6S(\alpha,\pi/3,\pi/6)$
Proof. For the first one, $\alpha = 0$ implies $k=1$, so
$$S(0,\beta ,\gamma ) = \sum\limits_{n = 1}^\infty {\frac{1}{{{n^2}}}( - \cos 2n\beta + \cos 2n\gamma )} + {\beta ^2} - {\gamma ^2} = \pi (\beta - \gamma )$$
For second one, the condition implies $k=0$. For third one, the condition implies $k=-1$, then just evaluate the series.
The fourth assertion is more crucial. We prove it carefully. By first bullet, it suffices to prove their derivative with respect to $\alpha$ are equal. When $\beta = \pi/3, \gamma = \pi/6$, one sees that
$$\frac{{1 + k}}{{1 - k}}\tan \alpha = \sqrt{3 \cos^2 \alpha-1} \sec \alpha $$
Since $\beta,\gamma$ are constants, $d\gamma = d\beta = 0$, $(3)$ shows
$$\tag{A}-\frac{1}{2}\frac{dS(\alpha,\pi/3,\pi/6)}{d\alpha} = \arctan(\sqrt{3 \cos^2 \alpha-1} \sec \alpha)$$
Now consider $S(\alpha,\pi - 2\alpha,\alpha)$, with $\beta = \pi - 2\alpha, \gamma = \alpha, d\beta = -2d\alpha$, $d\gamma = d\alpha$, one computes via $(3)$,
$$\tag{B}-\frac{1}{2}\frac{dS(\alpha,\pi - 2\alpha,\alpha)}{d\alpha} = 2 \arctan \left(\sqrt{\cos ^4\alpha-\cos ^2 2\alpha} \csc \alpha \sec \alpha\right)-2 \arctan \left(\sqrt{\cos ^4 \alpha-\cos ^2 2 \alpha} \tan 2 \alpha \csc ^2 \alpha \right)+2\pi $$
To complete the proof, it suffices to differentiate RHS of $6\times (A), (B)$, and see whether they are equal. This becomes a trivial but computational heavy task.
Now let $a>b$, consider
$$\sqrt {a - b} \int_0^x {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {b - t} (a - t)}}dt} = - 2\int_0^x {{{\tan }^{ - 1}}\sqrt t d({{\tan }^{ - 1}}\sqrt {\frac{{b - t}}{{a - b}}} )} $$
fix $\alpha = {\tan ^{ - 1}}\sqrt {\frac{{b - t}}{{a - b}}}$, we find $\beta,\gamma$ (depends on $a,b$ but not on $t$) such that $$\frac{{1 + k}}{{1 - k}}\tan \alpha = \sqrt t $$
One easily verifies one such pair $\beta,\gamma$ is
$$\gamma = {\tan ^{ - 1}}\frac{1}{{\sqrt a }}\qquad \beta = {\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} $$
Since $\beta,\gamma$ are independent of $t$, $(3)$ implies
$$dS(\alpha ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }}) = - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)d\alpha = - 2{\tan ^{ - 1}}\sqrt t d\alpha $$
Hence
$$\int_0^x { - 2{{\tan }^{ - 1}}\sqrt t d\alpha } = S({\tan ^{ - 1}}\sqrt {\frac{{b - x}}{{a - b}}} ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }}) + C$$
for a constant $C$ independent of $x$.
Set $x=0$, then one checks the third bullet point applies for $$S({\tan ^{ - 1}}\sqrt {\frac{b}{{a - b}}} ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }})$$
therefore it is $0$, hence $C=0$. Thus we proved
$$\tag{4}\int_0^x {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{(a - t)\sqrt {b - t}
}}dt} = \frac{1}{{\sqrt {a - b} }}S({\tan ^{ - 1}}\sqrt {\frac{{b -
x}}{{a - b}}} ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{
- 1}}\frac{1}{{\sqrt a }})$$
Now let $a=3, b=2$, we deduce from $(4)$
$$\int_0^1 {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = S(\frac{\pi }{4},\frac{\pi }{3},\frac{\pi }{6})\qquad \int_0^{5/3} {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6})$$
Now the second bullet applies for the former one, so
$S(\frac{\pi }{4},\frac{\pi }{3},\frac{\pi }{6}) = \frac{{{\pi ^2}}}{{48}}$,
this is $(1)$. For latter one, note that fourth bullet implies
$$6S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6}) = S(\frac{\pi }{6},\frac{{2\pi }}{3},\frac{\pi }{6})$$
but directly from definition (the $k$ associated with these two pairs are equal), one sees that
$$S(\frac{\pi }{6},\frac{{2\pi }}{3},\frac{\pi }{6}) - S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6}) = \sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{{{n^2}}}(\underbrace{\cos \frac{{2\pi n}}{3} - \cos \frac{{4\pi n}}{3}}_{=0})} + {(\frac{{2\pi }}{3})^2} - {(\frac{\pi }{3})^2} = \frac{\pi^2}{3}$$
thus $S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6}) = \frac{{{\pi ^2}}}{{15}}$, this is $(2)$. The integral claimed by OP is now established.