Integral $\int \arctan^2(x)\frac{dx}{x}$ $$\int \arctan^2(x)\frac{dx}{x}$$
I tried to use by parts , but ended up in a mess with the following : $$\int u^{2}\csc 2u \,\mathrm{d}u$$
 A: The integral has not a defined primitive. I will write you here the direct solution, considering that you may be (or you mayn't) familiar with a Special Function called the Poly-Logarithmic function
The result of your integration is:
$$\frac{1}{6} \left(6 i \tan ^{-1}(x) \text{Li}_2\left(e^{-2 i \tan ^{-1}(x)}\right)+6 i \tan ^{-1}(x) \text{Li}_2\left(-e^{2 i \tan ^{-1}(x)}\right)+3 \text{Li}_3\left(e^{-2 i \tan ^{-1}(x)}\right)-3 \text{Li}_3\left(-e^{2 i \tan ^{-1}(x)}\right)+4 i \tan ^{-1}(x)^3+6 \tan ^{-1}(x)^2 \log \left(1-e^{-2 i \tan ^{-1}(x)}\right)-6 \tan ^{-1}(x)^2 \log \left(1+e^{2 i \tan ^{-1}(x)}\right)\right)$$
where of course $i$ denotes the imaginary unit, and the writing $\text{Li}_n$ denotes the above quoted Poly Logarithmic special function, which you may be interested in.
References:
https://en.wikipedia.org/wiki/Polylogarithm
http://mathworld.wolfram.com/Polylogarithm.html
Some special integrations
You offered us an integral without limits. There is a cute special integral with a well know numerical solution:
$$\int_0^1 \frac{\arctan^2 x}{x}\ \text{d}x = \frac{1}{8} (4 \pi  C-7 \zeta (3))$$
Where $C$ is the Catalan constant and $\zeta(3)$ is the Riemann Zeta function of $3$, known also as the Apéry constant.
FYI: The integral becomes instead divergent if one (or both) of the extrema is $\pm \infty$.
Also the integrand function is odd which means
$$\int_{-a}^a\frac{\arctan^2 x}{x}\ \text{d}x = 0$$
A: As you indicate the integral given can be transformed via substitution, e.g. $$\int_0^a \arctan^n(x)\frac{dx}{x}=\frac{1}{2^n}\int_0^{2 \, tan^{-1}(a)} u^{n}\csc u \,\mathrm{d}u$$
and I find this second integral can be evaluated in terms of the infinite series shown below
$$\int_0^{x} u^{n}\csc u \,\mathrm{d}u={\frac{x^n}{n}}+\sum_{k=1}^\infty \frac{4 \,(2^{2k-1}-1)\,\zeta(2k)}{(2k+n)\,(2\pi)^{2k}} x^{2k+n} $$ 
This second integral (in a slightly different notation) can be found in G&R Table of Integrals etc 2.643.3 (page 221 in 8th Ed.)
The indefinite integral being
$$\int\arctan^n(x)\frac{dx}{x}={\frac{arctan^n(x)}{n}}+arctan^{n}(x)\sum_{k=1}^\infty \frac{4 \,(2^{2k-1}-1)\,\zeta(2k)}{(2k+n)\,\pi^{2k}} \,arctan^{2k}(x)
$$
