The given integral can be presented in the form of
$$I(\tau,a,b) = 4\int\limits_\tau^\infty \operatorname{arccot}^2\dfrac{\tau_2}a \,\dfrac{\mathrm d\tau_2}{\tau_2^2+b^2}.\tag1$$
Substitution
$$\varphi=\operatorname{arccot} \dfrac{\tau_2}a,\quad \tau_2 = a\cot\varphi,\quad\mathrm d\tau_2=-a(\cot^2\varphi+1)\,\mathrm d\varphi,$$
allows to write
$$I(\tau,a,b) = 4a\int\limits_0^{\operatorname{arccot}{\Large\frac\tau a}} \dfrac{\cot^2\varphi+1}{a^2\cot^2\varphi+b^2}\,\varphi^2\,\mathrm d\varphi
= 4a\int\limits_0^{\operatorname{arccot}{\Large\frac\tau a}} \dfrac{\varphi^2\,\mathrm d\varphi}{a^2\cos^2\varphi+b^2\sin^2\varphi}.\tag2$$
Then
$$I(\tau,a,a) = \dfrac4{3a}\varphi^3\bigg|_0^{{\Large\frac\pi2}-\arctan\Large\frac\tau a}
= \dfrac1{6a}\left(\pi-2\arctan\frac\tau a\right)^3\tag3$$
(see Wolfram Alpha test).
At the same time, the antiderivative of $(2)$ is
$$\color{brown}{\mathbf{\begin{align}
&J(\varphi,a.b)=4a\int \dfrac{\varphi^2}{a^2\cos^2\varphi+b^2\sin^2\varphi}\,\mathrm d\varphi =
\dfrac{2\varphi}b \big(\operatorname{Li_2}(r e^{2i\varphi})-\operatorname{Li_2}(^1\!/_{\large r}\, e^{2i\varphi})\big)\\[4pt]
&+\dfrac{i}{b}\big(\operatorname{Li_3}(re^{2i\varphi})-\operatorname{Li_3}(^1\!/_{\large r}\, e^{2i\varphi})\big)
+\dfrac{2i}{b}\varphi^2\ln\dfrac{1-re^{2i\varphi}}{1-\,^1\!/_{\large r}\,e^{2i\varphi}} +\operatorname{const},
\end{align}}}\tag4$$
(see also Wolfram Alpha calculations), where
$\operatorname{Li_n}$ is the Polylogarithm,
$$r=\dfrac{b-a}{a+b}.\tag5$$
Therefore,
\begin{align}
&\color{brown}{\mathbf{I(\tau,a,b)= J\left(\operatorname{arccot}\frac \tau a,a,b\right) -J(0,a,b).}}\tag6
\end{align}
If $a=11,\ b=17,\ \tau = 5,$ then $r = \frac3{14},$
$$I(\tau,a,b)\approx 0.10429\,46124\,85634,$$
(see also Wolfram Alpha result),
$$J\left(\operatorname{arccot}\frac \tau a,a,b\right)\approx 0.32355\,66131\,49807
-0.26227\,19119\,51703\,i$$
(see also Wolfram Alpha result),
$$J(0,a,b)\approx0.21926\,20006\,64173 - 0.26227\,19119\,51703\,i$$
(see also Wolfram Alpha result),
$$J\left(\operatorname{arccot}\frac \tau a,a,b\right)-J(0,a,b)\approx 0.10429\,46124\,85634\approx I(\tau,a,b).$$
Test results confirm obtained closed form for the given integral.