If one can prove that the given integral converges, it's not hard to compute its value. Let's assume from now that the integral does converge. Since $\tan^2(-bx)=\tan^2(bx)$ and $(-a)^2=a^2$, there is no loss of generality in assuming that $a,b>0$. Then
$$
I(a,b)=\int_0^\infty\frac{\ln\tan^2(bx)}{a^2+x^2}dx=2b\int_0^\infty\frac{\ln|\tan x|}{a^2b^2+x^2}dx=b\int_\mathbb{R}\frac{\ln|\tan x|}{a^2b^2+x^2}dx.
$$
Consider the function
$$
f: \mathbb{C} \to \mathbb{C},\ f(z)=b\frac{\ln|\tan z|}{a^2b^2+z^2}.
$$
Given $n \in \mathbb{N}$, with $0<1/n<ab<n$, we denote by $\Delta_n$ the bounded region of $\mathbb{C}$ whose boundary consists of the segment
$$
L_n=\{ x-\frac{i}{n}:\ |x|\le n\pi\}
$$
and the upper half circle
$$
\Gamma_n=\{\gamma_n(t)=-\frac{i}{n}+(n+\frac{1}{8})\pi e^{it}: \ 0 \le t \le \pi\}.
$$
The set of poles of $f$ that lie inside $\Delta_n$,
is $P=\{iab, k\pi/2:\ |k|\le 2n\}$.
For every $k$ with $|k|\le 2n$, $z_k=k\pi/2$ is a pole of order 2 with
$$
\text{Res}(f,z_k)=\lim_{z \to 0}\frac{d}{dz}(z^2f(z+z_k))=0,
$$
and since
$$
\text{Res}(f,iab)=\frac{1}{2ia}\ln\tanh(ab),
$$
we have
$$
\int_{\Delta_n}f(z)dz=i2\pi\text{Res}(f,iab)=\frac{\pi}{a}\ln\tanh(ab).
$$
Hence
$$
\int_{L_n}f(z)dz=\frac{\pi}{a}\ln\tanh(ab)-J_n
$$
with
$$
J_n:=\frac{\pi}{a}\ln\tanh(ab)-in\pi\int_0^\pi e^{it}f((n+\frac{1}{8})\pi e^{it}-\frac{i}{n})dt.
$$
Notice that
\begin{eqnarray}
|J_n|&\le&(n+\frac{1}{8})\pi\int_0^\pi|f((n+\frac{1}{8})\pi e^{it}-\frac{i}{n})|dt\cr
&\le& \frac{(n+\frac{1}{8})\pi}{((n+\frac{1}{8})\pi-\frac{1}{n})^2-a^2b^2}\int_0^\pi|\ln|\tan((n+\frac{1}{8})\pi e^{it}-\frac{i}{n})||dt\cr
&=&\frac{(n+\frac{1}{8})\pi}{((n+\frac{1}{8})\pi-1/n)^2-a^2b^2}\int_0^\pi\left|\ln\left|\frac{\exp(i(2n+\frac{1}{4})\pi e^{it}+\frac{2}{n})-1}{\exp(i(2n+\frac{1}{4})\pi e^{it}+\frac{2}{n})+1}\right|\right|dt\cr
&\le&\frac{(n+\frac{1}{8})\pi}{((n+\frac{1}{8})\pi-\frac{1}{n})^2-a^2b^2}A_n,
\end{eqnarray}
with
$$
A_n=\int_0^\pi |\ln|e^{i(2n+\frac{1}{4})\pi\cos t}e^{\frac{2}{n}-(2n+\frac{1}{4})\pi\sin t}-1|+|\ln|e^{i(2n+\frac{1}{4})\pi\cos t}e^{-(2n+\frac{1}{4})\pi\sin t+\frac{2}{n}}+1||dt.
$$
$A_n$ is clearly bounded, so we conclude that $J_n \to 0$ as $n \to \infty$, and
$$
I(a,b)=\lim_{n \to \infty}\int_{L_n}f(z)dz=\frac{\pi}{a}\ln\tanh(ab).
$$